Conversation
PTQ1_0 had only the generic scalar vec_dot on x86. Add SIMD kernels:
every 128-bit lane holds one source run of trit bytes times one power
of 3, so the unpack needs one in-lane shuffle (no VBMI). Trit t is
(3*v)>>8, computed with two byte compares; the dot uses t in {0,1,2}
and subtracts sum(y).
- AVX-512BW + VNNI: 64 weights per vector, dpbusd
- AVX2: one q8_0 sub-block per vector, dpbusd with AVX-VNNI or
AVX512-VL VNNI, else maddubs + madd
Other x86 CPUs keep the generic kernel.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Tested on an Intel laptop. This CPU takes the AVX2 + AVX-VNNI ( Setup: Intel Core Ultra X7 358H (Panther Lake, 16 cores, hybrid; AVX2 + AVX-VNNI, no AVX-512), Windows 11, MSYS2 UCRT64 GCC 16.2, Correctness
Performance, t=16
Thread sweep, tg32 (two runs each; t=16 understates the gain on this hybrid CPU):
Best-vs-best: about 2.0x decode on both models (27B: 2.13 vs 1.02 t/s; 2B: 18.5 vs 9.5 t/s) and about 2.7x pp64 on the 27B. Compared with the overlapping PRs on the same box (all three rewrite Not tested: the AVX2-without-VNNI ( Tested with Claude Code. |
|
Follow-up: MSVC builds and the AVX2-without-VNNI ( Setup: Core Ultra X7 358H (AVX2 + AVX-VNNI, no AVX-512), Windows 11, base =
Correctness of the fallback
Performance, t/s
(My first comment's GCC numbers are from before a stray background benchmark on this machine; the ones above are a clean rerun.) Tested with Claude Code. |
|
#248 merged first, so this conflicted with How the conflict was resolved: there's now a single Verified on a Core Ultra X7 358H, Windows 11:
The AVX2 rows match my pre-rebase KLD for this PR to the last digit, so the rebase didn't change behaviour. The What it adds over current Numerics note (raised by @AlexGabbia on #181): this kernel is not bit-exact with the generic one. It folds |
|
AVX-512 VNNI follow-up for the rebased branch (
The rebased branch stays as a reference for you to pull, @lenny76. We won't push it to this PR's branch. |
|
Test report for Setup
Results, t/s (median over 18 samples per arm)
With 8 threads, the four clean brackets all show +16-19% on generation. Within-run spread is small (MAD 0.08 base, 0.12 PR). Run-level slow state (not caused by this PR, but it affects measurements on this box). Some runs sit at about 10 t/s generation for all three repetitions, then the next run is normal:
Outside that state, the PR's run medians at Standalone kernel, same CPU (4096 rows x 5120, single thread, 48 interleaved samples):
Not tested here: KLD, the Happy to rerun other thread counts or configurations on this laptop if useful. |
|
Test report for the AVX2 tier of Setup
llama-bench, CPU only (
MTP on CPU (llama-server, 96 tokens, temperature 0, 24 threads, one run each): with the #181 kernel, Decode is memory-bound with this kernel on this box. I replaced the kernel with a stub that reads the same weight and activation bytes without decoding, alternating stub and real kernel per token inside one process. Median token time was 276 vs 283 ms at 16 threads and 279 vs 281 ms at 24 threads. The matmuls run at about 25 GB/s effective, so on single-channel RAM further compute savings in the kernel no longer show up in decode. On the slow-run state reported above: on this laptop, back-to-back runs drift from ~4.3 to ~2.5-3.7 t/s (sustained power/thermal), and the 24 workers often sit on only 17-20 distinct logical CPUs at the end of a graph ( Not tested here: KLD, the maddubs fallback, AVX-512, the exact |
4 participants
Overview
On x86,
ggml_vec_dot_ptq1_0_q8_0is aliased to the generic scalar kernel (arch-fallback.h), so PTQ1_0 runs at under 1 t/s on CPU. This PR adds two SIMD kernels inarch/x86/quants.c:dpbusd.dpbusdwhen AVX-VNNI or AVX512-VL VNNI is available (reusesGGML_DPBUSD_256), elsemaddubs+madd.Other x86 CPUs keep the generic kernel. Activations stay Q8_0, so numerics only change by float summation order.
How it works: the element order of a block (see
dequantize_row_ptq1_0) is trit nn ofqs[0..15]->16*nn + m, trit nn ofqs[16..23]->80 + 8*nn + m, trit nn ofqh[h]->120 + 2*nn + h. So every 128-bit lane of the unpacked weights is one source run times one power of 3, and the unpack is one in-lanepshufb(no VBMI needed, Cascade Lake has none). Per bytev = b*3^nn mod 256is twovpmullwplus a blend; the trit(3*v)>>8is(v >= 86) + (v >= 171); the dot uses t in {0,1,2} and subtractssum(y).Additional information
2x Xeon Gold 6262 (Cascade Lake), Linux, GCC 14. AVX-512 build:
-DGGML_NATIVE=ON. AVX2 build:-DGGML_NATIVE=OFF -DGGML_AVX2=ON -DGGML_FMA=ON -DGGML_F16C=ON -DGGML_AVX512=OFF(no zmm instruction inlibggml-cpu.so). Both runs include #249 (without it, token generation of any vec_dot type is limited on this machine by the src1 conversion, see that PR).Standalone kernel test (generic kernel copied verbatim as reference, random blocks, n = 128 ... 17408): results within 1.8e-8 relative error (float order only); one-hot activations recover every weight exactly (20000 cases, both kernels). Streaming n = 5120 rows:
Ternary-Bonsai-2-27B-PTQ1_0.gguf,llama-bench -ngl 0 -p 64 -n 32:Token generation is memory bound, so both kernels give the same speed there.
Correctness:
llama-perplexity -c 512 --chunks 8on an Italian novel: AVX2 23.4668, AVX-512 23.4743 (PQ2_0 on the same text: 23.5011). The generic kernel is too slow for a full run.dpbusdpath was only run through_mm256_dpbusd_epi32(AVX512-VL) in the standalone test; it has the same semantics as_mm256_dpbusd_avx_epi32.Requirements
🤖 Generated with Claude Code