ggml-cpu: add 2x2 register-blocked AVX2 kernel for ggml_vec_dot_q8_0_q8_0

[codspeed-hq]

What

ggml_compute_forward_mul_mat() calls the dot product once per (src0 row, src1 column) pair, so a Q8_0 matmul over an F32 activation re-loads every weight block - and re-converts every FP16 scale - once per column of the tile.

This adds an nrc == 2 path to the AVX2 ggml_vec_dot_q8_0_q8_0() that computes a 2x2 output tile (two src0 rows x two src1 columns) in one pass, and enables nrows = 2 for GGML_TYPE_Q8_0 on AVX2 (the ARM i8mm path already used nrows = 2; Q4_0/Q4_1/Q4_K/Q6_K already declare it too).

Why it wins

Very little of the per-block work is the multiply-accumulate itself. Per 32-value block a 1x1 dot product pays 2 loads, 2 FP16 scale conversions and 2 vpsignb (_mm256_maddubs_epi16() needs an unsigned first operand, so |x| and sign(x)*y must be materialized). In the tile, |x0|/|x1| and all four operands are shared across the four accumulators:

per 32-value block	4 x 1x1 dots	one 2x2 tile
32-byte loads	8	4
FP16 scale conversions	8	4
vpsignb	16	6
total instructions / 128 MACs	68	47

The four ymm accumulators and all operands stay in registers (no stack traffic).

Measured effect

CodSpeed CPU simulation, full macro suite (benchmarks/bench_macro.cpp), measured locally on the same machine for base and head:

Benchmark	Base	Head	Change
prompt_layer[q8_0]	1,007.3 ms	725.7 ms	+38.8%
lm_head[q8_0]	173.7 ms	124.0 ms	+40.1%

The other 11 macro benchmarks are unchanged. decode_layer[q8_0] generates a single token (ne11 == 1), so it keeps the 1x1 path.

For prompt_layer[q8_0] the instruction component drops (82.7% -> 81.5% of a 28% smaller total) and the cache component shrinks as each weight block is now read once for two columns.

Correctness

Results are bit-identical: each of the four sums uses its own accumulator, visits the blocks in the same order and applies the same d_x*d_y product as the 1x1 loop.

• FNV-1a hashes of the raw ggml_mul_mat output match an unmodified build over 105 Q8_0 cases (K in {32, 64, 256, 2048, 5632}, N in {33, 128, 2048}, M in {1, 2, 3, 8, 16, 17, 64}, 2-D and batched 3-D, with 1/2/8 threads); the 105 Q4_0 control cases are unchanged as well.
• The new path is genuinely exercised: injecting a deliberate error into the tile changes 40 of the 105 Q8_0 hashes (exactly the cases that satisfy the nr0 % 2 == 0 && ne11 % 2 == 0 conditions in ggml_compute_forward_mul_mat), and none of the Q4_0 hashes.
• ctest -L main passes (46/47; the only failure, test-tokenizers-ggml-vocabs, is a pre-existing environment issue - the git-lfs vocab files are not fetched in the sandbox).
• Compile-checked with no warnings for SSE4.2, AVX+F16C, AVX2+FMA+F16C, AVX2+AVX-VNNI and AVX-512+VNNI.

Scope

Only the AVX2 path in ggml/src/ggml-cpu/arch/x86/quants.c plus the nrows flag are touched, so this is an x86 change; the aarch64 kernel is untouched and unaffected. The tile only applies when the number of src1 columns is even, i.e. prompt processing and the LM head - single-token decode keeps the existing kernel.

[codspeed-hq]

Merging this PR will not alter performance

✅ 28 untouched benchmarks

---

_{Comparing codspeed-optim-2-2-register-blocked-avx2-kernel-for-ggml-vec-dot-1785415701799 (93f4a9a) with master (46819c9)}