Repository navigation
feat(TC-887): TCE1 segmented AES-256-GCM envelope - #322
Merged
Merged
Conversation
Adds tinycloud_core::at_rest with DekId, DataKey and ObjectNameKey (zeroized on drop, redacted Debug), the DataKeyProvider trait with DataKeyError, and a deterministic FixedKeyProvider for tests. Nothing in the node uses the module yet.
Contributor
Author
|
REQUEST_CHANGES: see TC-887 (reviewed head 641e2fe) |
FixedKeyProvider moves to at_rest/testing.rs, compiled only for this crate's tests or with the new test-util feature. Its Debug no longer prints the seed. Refresh test/m1-realdata-e2e/Cargo.lock for zeroize.
Adds the at_rest envelope: 64-byte TCE1 header, 64 KiB segments with counter nonces and a last-segment flag, per-blob HKDF-SHA256 key from the DEK and a random salt, AAD binding to the blob identity (kv-block, sql-checkpoint, sql-wal) plus the header. Whole-buffer seal/open in place, streaming EncryptReader/DecryptReader, plaintext-to-segment Range mapping, PayloadEncoding column flag, and NK-derived sealed object names. Known-answer vectors come from an independent Python reference. Nothing in the node calls the module yet.
EncryptReader now probes the source for EOF before sealing the final segment, so over-long input fails before a complete object is yielded. Enable aes/aes-gcm zeroize (aes-gcm >= 0.10.3). seal_with_header and EncryptReader::with_header are test-only. Document that header fields are unverified until a segment decrypts. Refresh the m1 lockfile.
samgbafa
force-pushed
the
skgbafa/tc-887-at-rest-envelope
branch
from
October 17, 2026 06:27
641e2fe to
7050fb8
Compare
Contributor
Author
|
REQUEST_CHANGES: see TC-887 (reviewed head 7050fb8) |
Add EnvelopeError::Unsupported for a well-formed later TCE generation (format_version dispatches on the first 5 bytes) and for payload encodings 2..=255. Mark EnvelopeError and DataKeyError non_exhaustive. seal reserves exactly ciphertext_len; DecryptReader trims without 32-bit overflow; docs say any segment authenticates the header and readers hold two segment buffers. Tests: edge-biased range proptest, empty-blob ranges, no output from a failing segment and sticky errors, range with trailing bytes, the 2^48 size limits, later generations. The Python reference now also emits length, range-mapping and negative vectors (checked by its own opener).
…lver guard SegmentRange keeps the ResolvedByteRange from map_range, so map_range holds the module's only ByteRangeSpec::resolve call; add that exact call to did_resolution_single_entry's allowlist like the storage backends' range calls.
Contributor
Author
|
APPROVE: see TC-887 (reviewed head 9966fe2) |
Correct the aes-gcm zeroize comment: polyval keeps H without a Drop, listed as an accepted residual with the hkdf PRK and hmac state. Note on EnvelopeError::Unsupported that callers must fail closed.
Contributor
Author
|
APPROVE: see TC-887 (head |
# Conflicts: # tinycloud-core/Cargo.toml # tinycloud-core/src/at_rest/key.rs # tinycloud-core/src/at_rest/mod.rs # tinycloud-core/src/at_rest/provider.rs
Contributor
Author
|
APPROVE: see TC-887 (head 3122c0a) |
This was referenced Oct 17, 2026
Merged
samgbafa
added a commit
that referenced
this pull request
Oct 17, 2026
…kfiles in CI (#332) * fix(TC-887): refresh the w5 lockfile and check separate workspace lockfiles in CI test/w5-policy-runtime-node-e2e/Cargo.lock lacked tinycloud-core's new aes, hkdf and zeroize dependencies and the aes/aes-gcm zeroize features (#320, #322), so cargo --locked failed there. Regenerated minimally. CI built only the main workspace and m1-realdata-e2e. A new lockfiles job resolves every fetchable workspace with --locked and runs scripts/ci/check-separate-lockfiles.py, which checks the w5 lockfile (its private git dependency is not fetchable in CI) for in-repo crates' direct dependencies and requested dependency features. * ci(TC-887): drop the hand-rolled w5 lockfile checker Keep the w5 lockfile fix and the --locked resolves of main, m1 and test/load/signer. w5 coverage follows separately with real cargo.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of TC-887 (encryption at rest). Follows #320 (merged).
origin/mainis merged in (3122c0a), so the diff vs main is the envelope only.What changed
New code in
tinycloud-core/src/at_rest/only. Nothing in the node calls it; no config, schema, storage or API change.envelope.rs: the TCE1 format. Header fields are documented as unverified hints until a segment decrypts."TCE" ‖ ASCII digit g ‖ byte gidentify the format generation (format_version); a well-formed later generation isEnvelopeError::Unsupported, not a header/integrity error.EnvelopeErroris#[non_exhaustive](andDataKeyError).TCE1, version 1, algorithm 1, flags,dek_id(u32),plaintext_len(u64), 32-byte random salt, segment size (2^16), zero padding.Header::parserefuses any unknown version, algorithm, flag, padding or segment size.ct ‖ tag16. Nonce =u32 BE index ‖ last-flag ‖ 7 zero bytes. Per-blob key = HKDF-SHA256(DEK, salt, SHA-256(context)). AAD = context ‖ header.BlobIdentity:KvBlock {space, content_hash},SqlCheckpoint {service, space, name, checkpoint_content_hash, checkpoint_size_bytes},SqlWal {service, space, name, revision, checkpoint_content_hash, delta_content_hash}.seal/openon aVec<u8>, in place.openrequires the header'sdek_idto match the key and the length to be exact, and never returns partial plaintext.ciphertext_len/plaintext_lensize conversions (the second for sizing sealed objects from a listing).stream.rs:EncryptReader/DecryptReaderoverfutures::io::AsyncRead, one segment buffered at a time.EncryptReaderseals the final segment only after the source reports EOF, so input longer than declared fails before a complete object is yielded.Header::map_range(ByteRangeSpec)resolves against the plaintext length and returns the ciphertext span to fetch;DecryptReader::rangedecrypts and trims it.names.rs:PayloadEncoding(NULL= legacy,1= TCE1,2..=255=Unsupported, anything else refused) andsealed_object_name/space_tag(v1/{space_tag}/{object_tag}, HMAC-SHA256 under the object-name key). Legacy vs sealed is decided by name namespace or column, never from the bytes;openon non-TCE1 bytes is an error.testdata/gen_vectors.py+vectors.json: an independent Python implementation (pycacryptography) written from the format spec, and the vectors it produces: seal vectors, an object-name vector, length conversions, 402 range mappings and 15 negative vectors that the Python opener itself rejects.tinycloud-node-server/tests/did_resolution_single_entry.rs: the resolver guard's allowlist gains the oneByteRangeSpec::resolvecall inat_rest/stream.rs::map_range(same treatment as the storage backends' range calls).tinycloud-coregains direct depshkdf 0.12,hmac 0.12(both already inCargo.lock) and dev-depproptest 1;aes-gcmis raised to0.10.3with itszeroizefeature, andaesgetszeroize(AES key schedule wiped on drop; the GHASH key, hkdf PRK and hmac state are not, noted in the manifest).test/m1-realdata-e2e/Cargo.lockrefreshed.seal_with_header,EncryptReader::with_header) are#[cfg(test)].How it's tested
cargo test -p tinycloud-core --lib at_rest: 30 tests (3 from #320).dek_id= u32::MAX) and one object-name vector, all produced by the Python reference. Rustseal,EncryptReader,openandDecryptReadermust reproduce or invert them byte for byte.Pendinginterleaved;plaintext_len(ciphertext_len(p)) == pand the converse; random Inclusive/From/Suffix ranges across segment edges equal the plaintext slice, and unsatisfiable exactly whenByteRangeSpec::resolve(plaintext_len)is; any single-byte flip anywhere is rejected byopenand the stream.dek_id; each malformed header byte; legacy SQLite/plain bytes;EncryptReaderinput shorter or longer than declared, and over-long input never releases the final segment (over_long_input_never_yields_the_final_segment, declared 0 … 2×64 KiB, read sizes 1/3/64 KiB+100; failed before the fix with 80 of 80 bytes released for declared 0); aSegmentRangefrom another header.length_vectors_match_reference(0 … 2^48, 2^48+1, u64::MAX; invalid ciphertext lengths),range_vectors_match_reference(402 specs over 7 lengths incl. 0 and k·64Ki±1, each satisfiable one also decrypted),negative_vectors_are_refused_like_the_reference(truncations, tag/body/salt/length flips, swapped segments, trailing byte, TCE2 header → same error kind whole-buffer and streaming).edge_biased_ranges_equal_plaintext_slice(lengths and endpoints at k·64Ki±1 and len±1),empty_blob_has_no_satisfiable_range,failing_segment_releases_nothing_and_errors_stick(output before the error is exactly the verified prefix, whole and range; next read fails),range_with_trailing_bytes_is_refused,size_limits_at_two_to_the_48,later_generations_are_unsupported_not_corrupt.PROPTEST_CASES=1500atd327906: 30/30 pass (138 s).cargo clippy -p tinycloud-core --all-targets -- -D warningsandcargo fmt --check: clean.python3 tinycloud-core/src/at_rest/testdata/gen_vectors.py > tinycloud-core/src/at_rest/testdata/vectors.json(no diff).Merge order: after the current main release is cut (TC-887 plan §7).