SIMD kernel throughput optimization: streaming ChaCha/Poly1305, fused AES AEAD rework, and bulk GHASH AAD#145
Merged
Conversation
ChaCha (x86_64 + i386): - Vertical kernels (AVX2 8-way, SSE2/SSSE3 4-way) now stream AGroups groups per call on a 64B-aligned frame: one-time prologue, light per-group state reload, resident c-pair rounds (2 c-stores + 2 c-loads per round), and a fused add-back/transpose/xor tail written straight to the output (no bounce buffer). - New i386 AVX2 8-way kernel (stack-resident state rows, resident rot16/rot8 masks) roughly doubles 32-bit throughput. - Engine bulk ladder: one WideGroupsSafe wrap guard + TryProcessWide per tier; each wide kernel is entered once per span instead of per group. Poly1305: - x86_64 AVX2 bulk loop is modulo-scheduled: r-power rows load into the freed temps while pending limbs absorb, h2 (folded early in the prior reduce) opens all five accumulators, and the next block's load + limb split thread through the multiply and lazy reduce. - i386 AVX2 and both SSE2 kernels: 64B-aligned frames, k-major multiply with the accumulator resident across its terms, mask26/padbit as memory operands (keeps H0..H4 resident on i386), and a one-time aligned in-frame table copy so SSE2 multiplies take memory operands. ChaCha20Poly1305: - Bulk data now runs one whole-span cipher call and one whole-span MAC call per ProcessBytes; the 512-byte stride loops are gone.
…M subsystem Mode-level: every AEAD mode now has a same-key re-Init fast path, so a fresh nonce per message no longer pays for key-dependent setup: - OCB: skip both key schedules, the L-table rebuild and the kernel re-acquire when the key is unchanged (keeps the Ktop cache; direction-aware). - CCM: cache the CTR wrapper and re-init it per packet with IV-only parameters; same-key Init skips the engine init and kernel re-acquire. Decrypt now writes straight into the caller's buffer and zero-wipes it before raising on a tag failure, instead of bouncing the whole plaintext through a scratch copy; the no-unverified-plaintext guard test asserts the new wipe contract. - GCM: skip the key schedule, hash subkey, multiplier and H-power table rebuild and the kernel re-acquire on same-key re-Init (the CTR keystream is direction-independent, so key equality alone gates it). Kernel-level: - CTR (both arches): the counter staging no longer read-modify-writes counter memory per block. The counter lives in registers native-endian across the span (the constant prefix is store-only), with carry-outs rare-pathed; this removes a store-forward stall per staged block. - OCB (both arches): round-interleaved, double-buffered offset ladder - each iteration's AES rounds stage the next iteration's deltas into the opposite ping-pong bank, so the ladder's serial bsf/load/xor chain rides the AES latency instead of running as a prefix. i386 is 6-wide (min 12 blocks), x86_64 8-wide (min 16) with a branch-free 64-bit counter and key memory-operand rounds. - GCM (both arches): the kernel now owns an H^8..H^1 table pre-multiplied by x in the reflected representation, so the per-batch 256-bit product reduces with two carry-less multiplies by the folded field constant instead of a long shift/XOR ladder, and the running GHASH state stays reflected across batches (byte-reversal only at kernel entry/exit). x86_64 additionally holds all loop-carried state in registers (no per-batch context memory traffic) and feeds round keys as aligned memory operands. i386 gets a 16-aligned frame: aligned accumulator slots, a resident byte-reverse mask used as a shuffle memory operand, in-register cross-product combining and an in-register counter.
Replace the per-block shuffle-port register inserts in the fused GCM kernel's counter build with aligned stack slots: the fixed 12-byte J0 prefix is staged once in the prologue, and each batch's AES-only round 9 stages the NEXT batch's four-byte counter words, so the loop-top movdqa loads never read a freshly written slot (a naive store-then-load form forwards-fails on the partial overlap and serializes the loop top - it measured 40% slower). The prologue pre-stages batch 0; the final batch over-stages into scratch. This removes sixteen shuffle-port uops per 128-byte batch and is worth 3-7% on x86-64. The i386 kernel keeps its insert-based build: the staged form measured neutral-to-negative there, with both placements tried. A Karatsuba variant of the batch GHASH (three carry-less multiplies per block plus a precomputed halves-XOR table column) was also implemented, verified and benchmarked: it is throughput-neutral on x86-64 and a 4-7% regression on i386 - the saved multiply is repaid on the same execution port by the added shuffle, plus an extra table load per iteration - so the schoolbook four-multiply form stays.
…ulk path The 4-/8-way GHASH batch kernels (GcmGhashFull) previously paid their whole setup on every 64/128-byte call: a ten-register nonvolatile save/restore, the byte-reversal round-trip of the running state, a long shift/XOR reduction and a Pascal call boundary. Worse, the GCM mode's AAD path never used them at all - ProcessAadBytes fed one block at a time through the interface-dispatched multiplier, running at ~110 MB/s. Kernels: the batch body now loops over an ABatchCount parameter inside the asm, so the save/restore, mask load, reduction constant and state byte-reversal are paid once per call; the running state stays reflected across batches (a register on x86-64, an aligned frame slot on i386, which also serves the byte-reverse mask as shuffle memory operands). The per-batch reduction is two carry-less multiplies by the folded field constant instead of the shift ladder. Table: TGcmUtilities.InitEightWayHPowFromH now emits every H power pre-multiplied by x (DivideP), which is what the folding reduction requires. Every consumer funnels through the same table and kernel body - the GCM 4-/8-way paths, the new bulk-AAD path, the GCM-SIV POLYVAL Horner batch and the fused CTR+GHASH kernel (whose constructor now takes a plain copy instead of shifting its own) - so the change is consistent across the board. The mode's post-dispatch scalar fallbacks (unreachable in practice: they share their gate with the table precompute) were rewritten as per-block multiplier folds, keeping them correct independent of the table layout. Dispatch: TryFusedFour/EightShuffledGhash and the POLYVAL batch chain gained the batch count; ProcessAadBytes hashes whole 128-byte spans in a single kernel call and leaves the remainder to the per-block loop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR re-architects the hottest x86 SIMD paths for ChaCha20-Poly1305, AES AEAD modes, and standalone GHASH, and adds same-key re-Init fast paths so per-message nonce rotation no longer pays full key setup.
ChaCha20 / Poly1305 streaming kernels
AGroupsgroups per call on a 64B-aligned frame: one-time prologue, light per-group state reload, resident c-pair rounds, and a fused add-back/transpose/xor tail written straight to output (no bounce buffer)WideGroupsSafewrap guard +TryProcessWideper tier; each wide kernel is entered once per span instead of per groupTChaCha20Poly1305bulk data now runs one whole-span cipher call and one whole-span MAC call perProcessBytes; 512-byte stride loops are goneAES AEAD mode fast paths (same-key re-Init)
Fused AES kernel rework
Standalone GHASH + bulk AAD
GcmGhashFullbatch kernels now loop overABatchCountinside asm (setup paid once per call, not per 64/128-byte chunk)TGcmUtilities.InitEightWayHPowFromHemits every H power pre-multiplied by x (DivideP) for the folding reductionProcessAadByteshashes whole 128-byte spans in a single kernel call; remainder falls back to per-block loop (AAD was ~110 MB/s via one-block-at-a-time dispatch)