Repository navigation
ChaCha20-Poly1305 on x86-64: seal_gather copies 32 or 64 bytes at a time - #1345
Merged
Merged
Conversation
vg_chacha20_poly1305_seal_gather copied its slices 16 bytes at a time through SSE2's movdqu, where Rust's memcpy, which copy + encrypt_in_place uses, copies a cache line at a time: encrypt was 5-9% slower than copying and encrypting in place. Each instance now copies as wide as the CPUs of the instance of vg_chacha20_poly1305_seal it calls allow (`Width`, from the implementation of vg_poly1305_blocks it calls): 64 bytes at a time with AVX-512's vmovdqu32 of zmm0, 32 with AVX's vmovdqu of ymm0, then 16 at a time with AVX's vmovdqu of xmm0 and vzeroupper, then a byte at a time; the baseline instance's code is unchanged. The proof of the copy (Gather/Copy.lean) is one loop lemma for any width (`chunks_ok`), from where the previous loop stopped, with a step lemma for each width's body; the rest of the proofs take the width as a parameter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ma2fA9rUTPXEEQjYyCMCRw
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
From the rustls provider analysis in #991. That provider wants to seal TLS records with
ChaCha20Poly1305::encrypt(gather, two pieces: payload and content type). It still copies and callsencrypt_in_place, becauseencryptmeasured 3–7% slower than that.Why it was slower
vg_chacha20_poly1305_seal_gathercopied its slices 16 bytes at a time with SSE2'smovdqu. Rust'smemcpy, which copy +encrypt_in_placeuses, copies a cache line at a time.Change
Each instance now copies as wide as the CPUs of the
sealit calls allow (SealGather.Width, taken from the variant'svg_poly1305_blocks):_avx512: 64 bytes at a time (vmovdqu32 zmm0)._avx2: 32 bytes at a time (vmovdqu ymm0).vmovdqu xmm0(not SSE2's, which would pay for the dirty upper half),vzeroupper, and the last bytes one at a time.No new instructions, no TCB or spec change, no Rust change.
Proof
Gather/Copy.leannow has one loop lemma for any width (chunks_ok), starting where the previous loop stopped, plus a step lemma per body (wideStep_ok,vexStep_ok).Loop,LoopCT,Fn,CTandVerifiedtake the width as a parameter.taint_decide, once per width.Measurements
Sapphire Rapids–class Xeon (this repo's development VM), two pieces (
n − 1and 1 bytes), minimum of 41 interleaved samples. Ratio isencrypttime over copy +encrypt_in_placetime.The 16-byte AVX2 row is noise; the copy isn't the cost there. The remaining ~2% (more at 16 bytes) is the wrapper itself: its frame, the second call, and a separate copy pass. A follow-up will remove that by XORing the keystream from the pieces straight into the output for short messages, so
encryptbeats copy +encrypt_in_placerather than trailing it.Recent CI benchmark runners for x86-64 have been AMD EPYC 7763 (no AVX-512), so the
_avx512instance can't be compared there.Checks
lake buildandEmit.lean, done on the revision the CI cache was built from: x86-64 TCB: modelmovq r64, xmm#1333 only adds a TCB instruction that no code uses, and generates nothing.check_lean_imports,check_lean_speed,check_variants,check_mcdt,check_arch_gates.VG_CPU_FEATURESunset, AVX2-only andnone.🤖 Generated with Claude Code
https://claude.ai/code/session_01Ma2fA9rUTPXEEQjYyCMCRw
Generated by Claude Code