ggml-cpu: add x86 AVX-VNNI dot product for PTQ1_0

[AlexGabbia]

ggml-cpu: add x86 AVX-VNNI dot product for PTQ1_0

PTQ1_0 currently has only the generic scalar vec_dot on x86 (arch-fallback.h aliases it with a "until a SIMD version lands" note). This adds an AVX2 + AVX-VNNI / AVX-512-VNNI implementation of ggml_vec_dot_ptq1_0_q8_0, following the same shape as the existing PQ2_0 kernel:

• decode the base-3 packed trits ((b * 3^p) & 0xFF) * 3 >> 8 to {0,1,2} codes with 16-bit lane arithmetic, no lookup tables
• dot(code - 1, qy) = dpbusd(code, qy) - dpbusd(ones, qy)
• per 32-wide q8_0 sub-block the integer sum and the float accumulation order match the generic kernel exactly

Validation

test-quantize-fns passes on clang and MSVC builds.

Ternary-Bonsai-2-27B-PTQ1_0.gguf (26.9B), CPU only, 16 threads, Core Ultra 9 275HX (Arrow Lake-HX, AVX-VNNI, no AVX-512), llama-bench:

build	generic	this kernel	speedup
MSVC 19.44 (the shipped Windows toolchain)	0.79 tg64 / 0.85 pp256	2.87 tg64 / 3.78 pp256	3.6x / 4.4x
clang 22	4.58 tg128 / 6.03 pp512	4.65 tg128 / 6.10 pp512	parity

MSVC does not auto-vectorize the generic decode loop, so the explicit kernel is a 3.6-4.4x decode/prefill speedup for the Windows release binaries. Under clang the generic auto-vectorizes to near-parity, so the kernel mainly pins the fast path independently of the compiler.

Notes

• Only the #if defined(__AVX512VNNI__) && defined(__AVX512VL__) || defined(__AVXVNNI__) builds take the new path; other x86 builds fall back to the generic implementation exactly as before.
• The arch-fallback.h alias for x86 is removed; the ARM and GGML_CPU_GENERIC aliases stay.

[Copilot]

🟡 Changes recommended

The new explanatory comment violates the repository's concise, non-hard-wrapped comment convention.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds an x86 VNNI-accelerated PTQ1_0/Q8_0 dot product while preserving the generic fallback.

Changes:

• Decodes packed ternary values with AVX2/VNNI intrinsics.
• Removes the obsolete x86 generic alias.

File summaries

File	Description
ggml/src/ggml-cpu/arch/x86/quants.c	Adds the optimized dot-product kernel.
ggml/src/ggml-cpu/arch-fallback.h	Enables x86-specific dispatch.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[AlexGabbia]

Thanks - the comment block is now two lines and keeps only the non-obvious VNNI transformation. The element order and the packing details are readable from the code below it and from dequantize_row_ptq1_0, so they do not need restating.

Verified again after the edit: test-quantize-fns passes (clang 22, AVX-VNNI build), no functional change.

[bri-prism]

Agent review: posted by the maintainer's coding agent at their request.

No findings in this source pass. The SIMD packing/sub-block mapping and generic fallback were inspected; the unsigned-code dot minus activation-sum correction preserves the intended integer dot.

Native x86/VNNI compilation and test-quantize-fns were not run here. The compiler-specific results in the PR description are contributor evidence, not an independent reproduction by this review.

Reviewed commit: 104bf3bafb1b2f89a108b8b5a7e8c8d0f33fc87a.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[AlexGabbia]

Adding the independent validation that the review notes was missing ("Native x86/VNNI compilation and test-quantize-fns were not run here").

Environment: Core Ultra 9 275HX (Arrow Lake-HX, AVX-VNNI, no AVX-512), Windows 11, MSVC 19.44 (cl.exe 14.44.35207), built with -DGGML_AVX_VNNI=ON.

• test-quantize-fns: passes, 0 failures, on both the patched tree and the unpatched one (same target set, 254/254).
• Object-level check: VPDPBUSD present 39 times in ggml-cpu/arch/x86/quants.c.obj in the VNNI build; absent (0 occurrences) in an otherwise identical AVX2-only build, which falls back to ggml_vec_dot_ptq1_0_q8_0_generic — so the new path is genuinely selected and the fallback is not optimized away.
• Bit-exactness harness on synthetic ternary blocks: 200 × 64 blocks, 0 mismatches SIMD-vs-generic and 0 SIMD-vs-reference.
• Integration: the patch applies cleanly on top of the cuda: branch-free PTQ1_0 MMQ tile loader and full Ampere tile table (2x prefill) #214-cuda: Bonsai 2 27B at the full 262k window with the MTP head on 12 GB Ada: #218 + #215 hybrid PTQ1_0 dispatch, in-place q4_0/q8_0 K/V flash attention (integration checkpoint) #221 stack (bonsai-combo, d8a7180) and builds there too; the stack currently needs the separate compile fix reported on CUDA: fast, batch-invariant small-batch PTQ1_0 mat-vec (1.5x decode on Ampere) #218 for any MSVC/GCC build.

Effect on the shipped Windows toolchain, Ternary-Bonsai-2-27B-PTQ1_0, CPU only, 16 threads: 0.79 → 2.87 t/s tg64 and 0.85 → 3.78 t/s pp256 with MSVC; with clang 22 the generic path already auto-vectorizes (4.58 → 4.65 t/s tg128), so the kernel mainly makes the fast path independent of the toolchain.

The Copilot note about the comment style is cosmetic; I can shorten that comment if you prefer.

[Copilot]

Copilot review overview

🟢 Approval recommended

The SIMD mapping, accumulation order, feature guards, and fallback path are consistent with the generic implementation.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Please keep this comment to the non-obvious VNNI transformation. The current ten-line prose block…

[AlexGabbia]

Merge-order note, in case it is useful: #248 adds an x86 vec_dot for PTQ1_0 as well, and it touches the same two places this PR does - the removal of the ggml_vec_dot_ptq1_0_q8_0_generic alias in the x86 block of arch-fallback.h, and a new ggml_vec_dot_ptq1_0_q8_0 in arch/x86/quants.c - so the two cannot both own the symbol and one of them has to rebase. They are complementary in content: #248 is SSE2/SSSE3 with a fall-through, this one is VNNI with a tail call to the generic, so the merged shape is #if VNNI ... #elif SSSE3/SSE2 ... #else generic, a small edit on whichever side lands second. I am not asking for priority: if #248 lands first I will rebase this one on top of it. Order aside, #248 does bring something this PR does not have, a dedicated per-pattern test for the ternary dot products; here the kernel is only exercised through test-quantize-fns' comparison against its double-precision reference.

[bri-prism]

Tested on an Intel laptop. This CPU takes the AVX-VNNI path.

Setup: Intel Core Ultra X7 358H (Panther Lake, 16 cores, hybrid; AVX2 + AVX-VNNI, no AVX-512), Windows 11, MSYS2 UCRT64 GCC 16.2, -DGGML_NATIVE=ON, CPU only (-ngl 0). Base = prism @ 3b19c377d; PR merged on top. llama-bench -p 64 -n 32 -r 2 -t 16, two rounds with the variant order reversed in round 2 (values are round 1 / round 2, t/s).

Correctness: matches the generic kernel, as claimed

• test-quantize-fns passes, including x86: SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0 #248's exact PTQ1_0 vec_dot check (dropped into this tree).
• Greedy 128-token output (latest-2B-PTQ1_0, temp 0) is byte-identical to the generic kernel.
• KLD vs base (llama-perplexity -c 512 --chunks 8, 2B PTQ1_0): max KLD 7.2e-5, same top token 100 %, RMS Δp 0.000 %.

Performance. Unlike the clang 22 parity result in the description, GCC 16 does not auto-vectorize the generic decode enough either, so the gain isn't MSVC-only:

model	base pp64	PR pp64	base tg32	PR tg32
Bonsai 2 27B PTQ1_0	1.70 / 1.67	2.70 / 2.66 (+59 %)	1.05 / 1.04	1.47 / 1.46 (+40 %)
2B PTQ1_0	40.26 / 27.21	57.61 / 42.60	8.03 / 7.08	9.13 / 8.91

(2B pp64 is noisy in both arms.)

Overlap: #250 and #248 rewrite the same function, so the three conflict. On this box, 27B PTQ1_0 pp64 / tg32 is #250 4.65 / 2.00, #248 (SSSE3, on its older base) 3.87 / 1.83, this PR 2.70 / 1.47. #250 is faster by about 1.7x prefill and 1.35x decode, but gives up exact order-matching (its KLD is still negligible: max 0.0022, 99.2 % same top). Worth deciding between them.

Not tested yet: an MSVC build (Build Tools 2022 is on the box; I can follow up with MSVC numbers).

Tested with Claude Code.

[bri-prism]

Follow-up with MSVC numbers, as promised. Your MSVC claim reproduces.

Setup: Core Ultra X7 358H (AVX2 + AVX-VNNI, no AVX-512), Windows 11. MSVC 19.44.35228 (VS 2022 Build Tools), -DGGML_NATIVE=OFF -DGGML_AVX=ON -DGGML_AVX2=ON -DGGML_FMA=ON -DGGML_F16C=ON -DGGML_AVX_VNNI=ON. I checked the binary: vpdpbusd present, no AVX-512. Base = prism @ 3b19c377d. llama-bench -ngl 0 -r 2, two rounds with the order reversed, 0 other llama processes during the run (logged). t/s:

model / test	MSVC generic	MSVC #181	speedup	GCC 16 generic (same session)
Bonsai 2 27B PTQ1_0, tg16, t=8	0.36 / 0.36	1.45 / 1.46	4.0x	0.89 / 0.88
2B PTQ1_0, tg32, t=4	3.61 / 3.62	14.69 / 14.66	4.1x	7.66 / 7.58
2B PTQ1_0, pp64, t=4	5.08 / 5.09	22.11 / 22.18	4.4x	13.32 / 13.20

That's in line with your 3.6x / 4.4x. MSVC's generic path is 2–2.6x slower than GCC's here, which confirms it doesn't vectorize the decode loop. test-quantize-fns passes the PTQ1_0 exact check in the MSVC build too.

For the choice between the overlapping PRs, same session and flags: #250 under MSVC is 2.97 (27B tg16), 31.6 (2B tg32) and 53.2 (2B pp64). That's about 2.0–2.4x this PR under MSVC. This PR's advantage remains exact agreement with the generic kernel's output.

Tested with Claude Code.

[AlexGabbia]

Kernel-level note on the three overlapping PRs, on one machine with all three kernels in the same binary.

I had posted merge-order advice on #181 earlier (tiers: #250 AVX2 / #181 VNNI / #248 SSE2). That was wrong, and the measurement says why. Setup: Core Ultra 9 275HX (Arrow Lake-HX, AVX2 + AVX-VNNI, no AVX-512), Windows 11, MSVC 19.44.35207, Release, single thread. Each binary links one PR's ggml_vec_dot_ptq1_0_q8_0 plus the tree's generic kernel, so speed and bit-agreement are measured on the same ruler. ns/call, 200k reps after warm-up:

K	#181	#250	#248
128	14.1	6.7	10.3
512	52.4	18.8	38.4
1536	154.3	53.3	111.0
5120	507.4	163.3	368.7
17408	1743.5	548.6	1237.7

The advantage is stable across three cache regimes and two sessions. Disassembly confirms each kernel is the real one (8 vpdpbusd in #181, 8 in #250, 20 vpmaddubsw + 11 pmaddwd in #248).

Two things I did not know when I posted the tier plan:

1. #250 already takes the VNNI path. arch/x86/quants.c:556-558 maps GGML_DPBUSD_256 to _mm256_dpbusd_avx_epi32 when __AVXVNNI__ is defined, and that sits inside its #elif defined(__AVX2__) tier. On this CPU — AVX2 + VNNI, no AVX-512 — #250 is the VNNI kernel, and it is 3.1× faster than #181's. So #181 is not a higher tier over #250; it is the same case, slower. Every CPU #181's guard accepts has AVX2, so it lands in #250's AVX2 tier regardless.

2. The bit-exactness claim is narrower than it looks. Comparing kernel against the generic one on 17500 real quantized inputs (nb = 1, 2, 4, 8, 20, 40, 80):

PR	differ from generic
#181	0 / 17500
#250	12233 / 17500 (70%), worst rel 8.2e-4
#248	0 / 17500

#250 pre-multiplies d0*d1 and uses _mm256_fmadd_ps, so it rounds in a different order — numerically harmless, but it breaks bit-reproducibility against the generic path. That is the property #181 had going for it.

But #248 has the same property and is 1.4× faster than #181, while also covering pre-Haswell x86-64 and PQ2_0. So #181 is dominated on both axes: #248 beats it on speed at equal exactness, #250 beats it on speed at the same CPU coverage.

Worth adding for the record: #250 also passes #248's new test_vec_dot_ternary. That test builds the packed weights from raw byte patterns at nb=1 and nb=3 and asserts an exact match, so it does not exercise the accumulation over more than a few blocks — all three kernels pass it. The divergence above appears only on real quantized data and larger nb. That is not a defect in the test, but it does mean the test does not decide between these three.

I am not asking anyone to keep this open on my account. On these numbers #248 + #250 cover the space and #181 adds no coverage: #248 for the SSE2 baseline and exactness, #250 for AVX2/AVX-512 speed. Closing #181 in favour of them is the right call, and the kernel is on record here and in #248's review thread if the exactness property is ever wanted back.

One caveat worth stating rather than hiding: this is single-threaded, and in the CPU decode path of Bonsai 2 27B token generation is memory-bound. The ranking is what I measured; the end-to-end effect on tg should be checked by whoever owns the CPU path.

Independent measurement, no affiliation with any of the three PRs.

[bri-prism]

Thanks for the careful measurement and for recommending this yourself. Agreed: with #248 merged and #250 covering AVX2/AVX-VNNI at about 3x this kernel's speed, #181 adds no coverage, so we're closing it in favour of those two. Your exactness analysis is a useful record if bit-reproducibility against the generic path is ever needed. Thank you.