[alex]

Stacked on #1350 (which is stacked on #1331). This is a Rust-only change.

This comes from the rustls provider analysis in #991, the second step on #1228. The provider seals TLS records with AesGcm::encrypt from two pieces (the payload and the content-type byte). The goal is that encrypt is never slower than copying into the output and calling encrypt_in_place.

What changes

AesGcm::encrypt now copies the pieces into out and encrypts there in place when the text is shorter than its backend's threshold (Backend::copy_below). The copy uses memcpy, and the in-place encryption goes through a new internal seal that encrypt_in_place now shares, so the lengths aren't checked twice. Calling vg_aes_gcm_seal_gather has a fixed cost (the descriptors, its argument frame and work slots, then its call of seal or of the streaming functions) that exceeds a memcpy for short texts.

Thresholds, measured on a VAES/AVX-512 Xeon:

• VAES + VPCLMULQDQ + AVX-512 (out-of-place interleaved loops): (3072, 16384). The gather pays for itself from about 3 KiB when every piece but the last ends on a block boundary. When one doesn't, the streaming functions buffer the partial block, which costs about 100 ns more, and the gather only catches up from about 16 KiB. TLS payloads are arbitrary lengths, so the unaligned case is common.
• Every other backend: (65536, 65536). Their gather copies the text too (x86-64 AES-GCM: seal_gather copies a short text and seals it in place #1350), so copying here costs less at any length. The 64 KiB cap is above the largest TLS record. It keeps vg_aes_gcm_seal_gather in use, and tested, for longer texts, where the difference is negligible.

The match is exhaustive over the x86-64 backends, like Backend::prepared. Other architectures still always gather.

Measurements

The setup is encrypt of two pieces (n bytes, then 1 byte), 13-byte AAD, warmed key, minimum of 33 interleaved samples, in ns. "Before" is #1350 and "after" is this PR.

"copy+ip" copies the pieces into the output and calls encrypt_in_place, in a non-inlined function taking slices, as a caller like rustls would. Inlined, with the nonce and AAD lengths known at compile time, it measures 10–15 ns faster at small sizes; real callers don't have those lengths at compile time.

VAES + VPCLMULQDQ + AVX-512

n	copy+ip	before	after	after / copy+ip
16	80.1	85.3	81.3	1.015
128	87.9	89.8	87.8	0.999
256	108.2	127.7	110.1	1.018
384	122.6	141.7	126.6	1.033
1024	207.3	212.1	207.0	0.999
3071	353.6	450.3	354.4	1.002
3072	382.4	352.3	353.1	0.923
4095	422.2	515.6	425.9	1.009
4096	450.7	412.2	413.8	0.918
8191	737.4	795.9	740.2	1.004
8192	765.9	708.2	706.3	0.922
16383	1324.2	1311.9	1313.1	0.992
16384	1354.7	1225.6	1230.1	0.908

AES-NI + PCLMULQDQ + AVX (VG_CPU_FEATURES=aes,pclmulqdq,ssse3,avx)

n	copy+ip	before	after	after / copy+ip
16	117.8	119.5	119.7	1.016
511	234.6	246.5	238.2	1.015
2048	566.1	563.8	555.4	0.981
4096	940.5	941.7	939.3	0.999
16383	3110.7	3174.2	3109.3	1.000
16384	3148.6	3205.4	3147.3	1.000

I measured these before splitting out the internal seal. After the split, AVX-512 at 0–511 bytes measures 0.92–1.03 of copy+ip. The other backends (VAES + VPCLMULQDQ on AVX2, AES-NI + PCLMULQDQ) measure 0.97–1.01 of copy+ip at every size.

So encrypt now matches copy+ip on every x86-64 backend. On AVX-512 it is about 8–9% faster for block-aligned payloads from 3 KiB. Being faster on the other backends needs out-of-place interleaved loops for them. Being faster on AVX-512 for unaligned payloads needs the streaming path to handle a block split between pieces without buffering it. Both are follow-ups.

Interaction with #1331's two-slice path

#1331 now also has seal_two_slices_prepared. It covers a prepared AVX-512 key with a 96-bit nonce, at most 16 bytes of AAD, and a piece of at least 256 bytes followed by at most one byte, which is exactly the rustls record shape. encrypt tries it first and falls back to the copy above. I measured both orders on the same Xeon, with the same two-piece setup (minimum of 45 samples, in ns):

n	two-slice first	copy first	copy first / two-slice first
512	134.9	173.8	1.288
768	147.3	209.4	1.422
1024	172.5	218.0	1.264
1500	267.4	327.4	1.224
2000	301.3	360.9	1.198
3000	367.7	435.4	1.184
511	201.7	144.3	0.715
767	218.9	164.0	0.749
2047	309.3	267.3	0.864

The two-slice path first is 18–42% faster everywhere except where n % 16 == 15. There the byte completes the last block, and #1331's path takes its separate tail call. That case belongs to #1331's code, so it is not special-cased here. For texts the two-slice path doesn't take (short texts, other shapes, other backends), the tables above stand.

Tests

copy_below covers every backend the CPU has, on each side of both thresholds: one piece, a 16-byte piece then the rest, and a 1-byte piece then the rest. Each case is compared against the baseline's in-place encryption. The test is in the aes_gcm CPU group by module path, so every CI CPU runs it. #1331's prepared_gather_tails is kept alongside it.

Validation

cargo test with Wycheproof on the default backend, with VG_CPU_FEATURES=aes,pclmulqdq,ssse3,avx, and with VG_CPU_FEATURES=none; cargo fmt --check; cargo clippy --all-targets -- -D warnings; ci/check_variants.py; ci/check_arch_gates.py; ci/cpu_tests.py lint. Rerun after restacking on #1331's two-slice commit.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FoYWDDEDFq1DFev5X7n2aq

[gaynor-anthropic]

This seems like the wrong fix, why are we not speeding up the encrypt op?

[alex]

Are you going to address my feedback?

[alex]

You're right, and I'm sorry I didn't answer sooner: your comments never reached my session, and I should have checked the thread. This PR adds dispatch in Rust around a slow function rather than making the function fast, so I'm closing it.

What replaces it: AesGcm::encrypt stays a thin call into vg_aes_gcm_seal_gather, and the work goes into the assembly, one PR per piece:

1. Out-of-place interleaved AES/GHASH loops for every x86-64 backend, so encrypting from the pieces never copies. The VAES + VPCLMULQDQ (AVX2) loop is written and proven. Locally it is 6–14% faster than copy + encrypt_in_place at 4–16 KiB and 22–24% faster at 64–128 KiB. The AES-NI + AVX loop is next.
2. Lower fixed per-call overhead in seal_gather: the descriptor walk, the argument frame and work slots, and the extra call layer.
3. A short-input path inside the gather code itself.
4. A block split across pieces handled in the loop, instead of buffered by the streaming functions; buffering costs about 100 ns today.
5. The AVX-512 remainder of 1–15 blocks after the 16-block loop, which costs 25 ns + 5 ns per block today.

Each PR will include its measurements.

---

Generated by Claude Code