designs/storage: client-side encryption and key management - #4
Open
ilchu wants to merge 1 commit into
Open
Conversation
Add the design for Web3 Storage V2 client-side encryption: a per-drive key hierarchy, deterministic per-chunk encryption that keeps content-defined chunking useful on ciphertext, and drive sharing via an on-chain X25519 key registry plus HPKE wraps. Diagrams sit next to the README; open items are collected under Unresolved Questions. The storage index entry gains a pointer to the new design.
franciscoaguirre
left a comment
franciscoaguirre
left a comment
Contributor
There was a problem hiding this comment.
First review pass, will do more
|
|
||
| ## Summary | ||
|
|
||
| Web3 Storage promises that storage providers hold user data without being able to read it. Version 1 of client-side encryption delivers a sound cipher but nothing around it: a single raw symmetric key encrypts each file as one opaque blob, the Rust SDK and the browser UI produce mutually unreadable formats, one of the two browser apps doesn't encrypt at all, and there is no key rotation, no recovery, and no way to share encrypted data with another account. Separately, [PR #209](https://github.com/paritytech/web3-storage/pull/209) introduces content-defined chunking (CDC). While useful for interactive applications such as HackM3, whole-blob encryption with random nonces nullifies CDC: a one-byte edit to an encrypted file changes every byte of the ciphertext, so every version is stored at full size and nothing ever deduplicates. |
Contributor
There was a problem hiding this comment.
It's not really a promise that we encrypt, it's up to the use-case. We could have unencrypted buckets
| | | | | ||
| | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| | **Start Date** | 2026-07-21 | | ||
| | **Description** | Replaces Web3 Storage's V1 whole-blob encryption with a per-drive key hierarchy, deterministic per-chunk encryption that keeps content-defined chunking useful on ciphertext, and drive sharing via an on-chain key registry and HPKE. | |
Contributor
There was a problem hiding this comment.
When you say per-drive you mean per-bucket, right?
|
|
||
| 1. **A key hierarchy.** One long-lived master key per drive (the Key Encryption Key, KEK), from which every other key is derived on demand. Nothing but wrapped copies of the KEK is ever stored. | ||
| 2. **Encrypted CDC.** The client — not the provider — chunks the plaintext, and encrypts each chunk with a key derived from the drive KEK *and the chunk's own content hash*. This makes encryption deterministic per chunk: the same piece of content always produces the same ciphertext within a drive, so the provider's existing content-addressed deduplication keeps working on encrypted data, and small edits re-upload only the changed chunks. | ||
| 3. **Sharing.** An on-chain registry maps accounts to encryption public keys; granting someone access to a drive means encrypting ("wrapping") the drive KEK to their public key. No data is re-encrypted, no secret travels out of band, and the provider learns nothing. |
Contributor
There was a problem hiding this comment.
What do you mean by this? Isn't this just encrypting to that account's public key? We don't need an "on-chain registry"
Also, just giving them the key means they have access to everything in the bucket, right? What if we just want to share one file? What if we want to share everything up to this point but not future items?
| What currently works in Web3 Storage for client-side encryption: | ||
|
|
||
| 1. The Rust SDK implements XChaCha20-Poly1305 encryption (format version `0x01`) with a versioned wire format `[version byte][24-byte nonce][ciphertext + tag]`. Fresh random nonces per encryption, key material zeroed on drop, good unit and integration coverage (round-trips, wrong-key, tampering, version checks, nonce uniqueness). | ||
| 2. Encryption runs before upload and chunking, so the provider only ever sees ciphertext — the core zero-knowledge property holds for data that *is* encrypted. |
Contributor
There was a problem hiding this comment.
"the core zero-knowledge property"? What's the goal of this bullet point?
| 4. **No key management exists.** No password-based derivation, no key wrapping, no rotation, no sharing, no recovery. Losing the key loses the data; this is documented V1 scope, but it is also the wall every planned feature hits. | ||
| 5. **Encrypted partial reads always fail.** The AEAD tag covers the whole blob, so range reads of encrypted data are impossible by construction (a separate bug also mis-trims ranged reads of chunked plaintext). | ||
| 6. **The browser decides whether to decrypt by sniffing the first byte** of downloaded data against `0x02`. There is no stored per-object "this is encrypted" flag, so wrong-key and not-encrypted failures are indistinguishable to users. | ||
| 7. **No browser-side tests** cover encryption. |
Contributor
There was a problem hiding this comment.
This is irrelevant to this design. We could have tests without the goal of this design which is chunk before encryption and better handle keys
| 1. One blob format decryptable in both Rust and browser environments without the version split. | ||
| 2. Encrypted data retains CDC benefits: cheap edits, within-drive dedup, and ranged / resumable reads. | ||
| 3. Key rotation without re-uploading data (rewrap keys, not re-encrypt content). | ||
| 4. Drive sharing between accounts — first and foremost granting *read* access to non-owners (readers of a drive who must not thereby gain write rights) — with no provider trust and no out-of-band secret exchange. |
Contributor
There was a problem hiding this comment.
Why is "out-of-band secret exchange" a bad thing?
|
|
||
| ## Stakeholders | ||
|
|
||
| Affected: users of the Rust and TypeScript SDKs and of the Drive and S3 browser UIs; provider-node operators (a second upload lane and keystore endpoints); the runtime (one new storage map and extrinsic); applications built on the storage, such as HackM3; and PR #209, whose upload seam this design plugs into. |
Contributor
There was a problem hiding this comment.
The stakeholders of this design are the storage team itself and the applied engineering team because of how they might use Capacity. That's it. No need to mention stakeholders of the apps since no one is using them right now. The runtime is not a stakeholder.
Have you spoken to Applied Engineering about how they usually handle client-side encryption in their apps? They might not have the requirement of getting CDC to work but key management is something they must have dealt with
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
designs/storage/web3-storage-encryption/— the design for Web3 Storage's V2 client-side encryption: a per-drive key hierarchy (KEK + HKDF-derived keys), deterministic per-chunk encryption that keeps content-defined chunking (PR paritytech/web3-storage#209) useful on ciphertext, and drive sharing through an on-chain X25519 key registry plus HPKE wraps. Diagrams sit next to theREADME.md. The storage index entry (designs/storage/web3-storage.md, from #3) gains a pointer to it.Open items are collected under Unresolved Questions rather than left inline, per the README.
Implementation phasing
Kept out of the design per "plans belong in issues and PRs":
@noble/ciphersin TS), version-sniff on download, add a per-object encrypted flag (replacing first-byte guessing), bring the Drive UI to encryption parity, add browser-side tests. No dependency on anything below; pure debt payment and user trust.0x03(large, the bulk of work). Keyed FastCDC client-side (Rust + TS/WASM), per-chunk keyed-convergent encryption, encrypted manifests, provider chunk-mode upload + keystore endpoints, browser performance benchmark (WASM go/no-go, cipher decision per Alternative G). Coordinate with and land after web3-storage#209.set_encryption_keyextrinsic and storage, HPKE wrap/unwrap in both SDKs, share/revoke/rotate flows, lazy-revocation semantics documented user-facing. Depends on Phase 1 and a runtime release.Rollout is additive throughout, with the per-phase dependencies above, and preserves V1 compatibility (
0x01/0x02stay readable).