Vertical ChaCha/Poly1305 SIMD kernels, IBulkStreamCipher, and unified bulk ProcessBlocks(count) APIs#141
Merged
Merged
Conversation
…tream tests Kernels / engines: - Replace the row-layout ChaCha AVX2 kernels with vertical (word-per-lane) kernels: SSE2/SSSE3 4-way (256B) and AVX2 8-way (512B), on x86_64 and i386; retire the old row kernels. - Add TChaChaBaseEngine as the shared ChaCha core (block function + the 512/256/128/64B dispatch ladder). TChaCha7539Engine and the legacy DJB TChaChaEngine derive from it and differ only in counter width: one CHACHA_CTR64-parameterized kernel serves both variants (DJB advances the 64-bit counter; a WideBlocksSafe guard drops a tier near the low-word wrap). IBulkStreamCipher: - Add ClpIBulkStreamCipher, the stream analog of IBulkBlockCipher. TSalsa20Engine and TChaChaBaseEngine implement ProcessBlocks(count) via a virtual DoProcessBlocks ladder (Salsa 2->1, ChaCha 8->4->2->1) with pointer-form *Fast inner helpers. - TChaCha20Poly1305 resolves it and processes the AEAD body 512 bytes (8-way) per call instead of the interface-capped 256, keeping the same running-count bookkeeping. ~40% AEAD throughput on large buffers (16MB: 690 -> 982 MB/s). Tests: - Add TStreamCipherTestBase (engine-factory + label hooks, AES-test pattern) with reusable whole-vs-chunked, ProcessBytes-vs-ReturnByte and bulk-vs-single-block checks. Reparent the Salsa20 / XSalsa20 / ChaCha (DJB) / ChaCha7539 / XChaCha20 suites onto it, keeping their KAT vectors and dropping the duplicated per-suite harnesses.
Now that IBulkStreamCipher.ProcessBlocks(count) exists, the fixed-width bulk
methods are redundant with the unified DoProcessBlocks ladder.
- Engines: TChaChaBaseEngine and TSalsa20Engine ProcessBytes/DoFinal compute the
whole-block count and make a single DoProcessBlocks call (which ladders
8->4->2->1 / 2->1 internally) plus the partial tail, replacing the fixed-width
tier cascades (and their per-tier re-validation). Remove the public
ProcessBlocks2/4/8 (ChaCha) and ProcessBlocks2 (Salsa).
- Interface: drop ProcessBlocks2/4 from IChaCha7539Engine (it now exposes only
DoFinal + ProcessBlock); bulk is purely the optional IBulkStreamCipher
capability. Engine bulk surface is {ProcessBlock, ProcessBlocks(count)}.
- TChaCha20Poly1305: collapse the ProcessBlock/2/4/8 wrappers into one
ProcessBlocksBulk(count) and drive the AEAD body through IBulkStreamCipher as a
512B (8-way) stride loop + a single remaining-blocks call, keeping the same
FDataCount bookkeeping. A single-block else-path covers a custom engine that
exposes no bulk interface.
…sBlocks(count) The fixed-width batch methods on IAesHardwareEngine had no production caller (all modes use IBulkBlockCipher.ProcessBlocks(count); the fused AES-NI kernels use the raw key schedule) and baked x86's 4/8 widths into what is meant to be an architecture-neutral hardware-AES interface (a future ARMv8 engine seats on the same interface). - IAesHardwareEngine: drop ProcessFourBlocks / ProcessEightBlocks (array + ptr); it now adds only the single-block ProcessBlock(ptr) over IBulkBlockCipher. Bulk is the width-agnostic ProcessBlocks(count) ladder each engine drives at its own internal widths. - TAesEngineX86: remove the public fixed-width methods and their partial-overlap stack-buffer staging (no caller; it contradicts the identical-or-disjoint IBulkBlockCipher contract). The 4/8-wide *Fast helpers stay as the strict-private inner steps of the ProcessBlocks ladder; single-block ProcessBlock keeps its overlap staging. - AesHardwareEngineTests: migrate batch-vs-single and PByte-parity coverage to ProcessBlocks(4)/ProcessBlocks(8) (arch-neutral, ready for a future ARM engine); drop the 6 partial-overlap memory-layout tests that exercised the removed staging.
Fills the sub-AVX2 gap in the Poly1305 dispatch ladder: without it, any non-AVX2 CPU fell back to the radix-2^26 scalar path, collapsing the MAC (~240 MB/s) and Poly-bound AEAD (~70 MB/s). The new kernel is a 2-lane (xmm) radix-2^26 narrowing of the 4-way AVX2 kernel — same 5x5 schoolbook multiply mod 2^130-5 with the 5*r wraparound, 2 blocks/iter against an r/r^2 power table, pshufd horizontal fold — keeping SIMD/scalar radix parity across the whole ladder.
- x86_64: the bulk-loop carry-reduce is restructured into two interleaved chains (low D0->D1->D2->D3->H4, high D3->D4->H0->H1) to shorten the serial critical path. Measured ~+2.5% MAC / ~+2% AEAD; the run-once tail and final reduces stay serial. - i386: the reduce stays serial. The two-chain schedule was benched ~4-12% SLOWER on i386 -- its 8-ymm register file (D0..D4 plus an H backup spill to a 320-byte esp frame, and the multiply spills each accumulator per term) is throughput-bound, and the second carry temp would clobber padbit, forcing a rebuild each iteration. Both arches emit identical MACs. Both AVX2 kernels are re-emitted with canonical minimal-displacement VEX encodings (~20% fewer bytes than before) and plain mnemonic comments. Verified by KAT on both arches plus a displacement-insensitive byte diff against the previous kernels: i386 matches instruction-for-instruction; x86_64 differs only in the reduce and in disp widths.
The rot16/rot8 rotations used vpshufb ymm, ymm, [rsp+mask], reloading the shuffle control from the stack on every quarter-round - ~160 mask loads per 512-byte block, all landing on the load ports that the c-word spills already contend for. At each rotate step the quarter-round temp registers are free, so load the mask into a temp once per rotate phase and use vpshufb ymm, ymm, ymm for all four lanes. The pshufb count (and its port pressure) is unchanged, but mask loads drop to ~40 per 512-byte block - a strict reduction of ~120 load micro-ops per block, relieving the load ports the c-spills bottleneck on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR overhauls x86 ChaCha and Poly1305 SIMD paths, introduces a bulk stream-cipher capability interface, and unifies fixed-width bulk APIs across stream and block engines.
ChaCha SIMD + shared engine core
TChaChaBaseEngineas the shared ChaCha core (block function + 512/256/128/64B dispatch ladder);TChaCha7539Engineand legacy DJBTChaChaEnginederive from it and differ only in counter widthCHACHA_CTR64-parameterized kernel serves both variants; DJB advances the 64-bit counter with aWideBlocksSafeguard near the low-word wrapIBulkStreamCipher+ ChaCha20-Poly1305 AEADClpIBulkStreamCipher— the stream analogue ofIBulkBlockCipher; resolved viaSupports()and driven throughProcessBlocks(count)TSalsa20EngineandTChaChaBaseEngineimplementProcessBlocks(count)via a virtualDoProcessBlocksladder (ChaCha 8→4→2→1, Salsa 2→1) with pointer-form*Fastinner helpersTChaCha20Poly1305resolves bulk capability and processes the AEAD body in 512-byte (8-way) strides instead of the interface-capped 256B pathPoly1305 SIMD
API cleanup — width-agnostic bulk
ProcessBlocks2/4/8intoProcessBlocks(count); engines compute whole-block count and ladder internallyProcessBlocks2/4fromIChaCha7539Engine; bulk is purely the optionalIBulkStreamCiphercapabilityProcessFourBlocks/ProcessEightBlocksintoProcessBlocks(count)onIAesHardwareEngine; remove unused fixed-width staging fromTAesEngineX86Tests
TStreamCipherTestBase(engine-factory + label hooks) with reusable whole-vs-chunked,ProcessBytesvsReturnByte, and bulk-vs-single-block checksChaCha7539ProcessBlocks2TestsProcessBlocks(4)/ProcessBlocks(8)