Conversation
Three Vulkan backend optimizations for Bonsai 2 / PTQ1_0 class models, developed and measured on a Radeon 890M iGPU (Windows, MinGW build): 1. PTQ1_0 trit decode via lookup table (3^n mod 256 has period 64) The scalar recurrence loop ran up to 119 multiply-mod iterations per weight. Since 3^64 == 1 (mod 256), a 64-entry constant table replaces the loop with one multiply + one lookup. Bit-identical output. Applied to both consumers: shared ptq1_0.glsl header (mul_mat_vec, get_rows, copy_from_quant, mul_mm) and the standalone dequant shader. 2. Vectorized dequantize4 for PTQ1_0 The codec layout guarantees that any 4-aligned element group shares one recurrence exponent and reads consecutive bytes, so the whole group decodes with one branch + a vector multiply instead of four full branch chains. Bit-identical output. 3. MUL(signs) + FWHT fusion (mirrors ggml_metal_op_fwht_signed) Folds the Hadamard sign-flip MUL into the FWHT kernel's load pass via dedicated fwht_signed_* pipelines (3 descriptors). The matcher scans forward over empty reshape ops to the FWHT-hinted MUL_MAT, validates the transform width from the matmul operand (not the flattened activation width, which may be a concat of many transforms), and the hinted matmul is marked to avoid a second dispatch of the transform. Base fwht pipelines are untouched (2 descriptors, unchanged layout). Measured on Ternary-Bonsai-2-27B (PTQ1_0 variant), Ryzen AI 9 HX 470 iGPU, -ngl 99, single sequence decode: PTQ1_0 tokens/s: 1.85 -> 3.26 (+76%) from (1) + (2) Q2_0 band unchanged at ~8 t/s (uses standard Q2_0 kernels; (3) is speed-neutral on this driver but removes one kernel launch + one full activation read/write per Hadamard site, matching the motivation of the Metal implementation). Correctness: 7/7 generation checks (arithmetic, facts, translation, code) pass identical to baseline; decode output is bit-identical for (1) and (2) by construction. Authored with AI assistance (disclosed per CONTRIBUTING.md AI policy); every line reviewed and understood by the submitter.
There was a problem hiding this comment.
🟡 Changes recommended
Fusion bookkeeping, unavailable pipelines, and misaligned sign buffers can cause invalid dispatches or incorrect results.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Optimizes Vulkan PTQ1_0 decoding and introduces fused sign multiplication with FWHT.
Changes:
- Replaces serial trit recurrence with lookup tables.
- Vectorizes four-element PTQ1_0 decoding.
- Adds signed FWHT shaders, pipelines, and graph fusion.
File summaries
| File | Description |
|---|---|
vulkan-shaders-gen.cpp |
Generates signed FWHT variants. |
ptq1_0.glsl |
Adds table-based trit decoding. |
fwht.comp |
Applies optional sign vectors during FWHT loads. |
dequant_ptq1_0.comp |
Accelerates standalone PTQ1_0 decoding. |
dequant_funcs.glsl |
Vectorizes four-element PTQ1_0 decoding. |
ggml-vulkan.cpp |
Registers pipelines and implements fusion dispatch. |
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 7
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Require the signed FWHT pipeline to exist before fusing (null on some devices) - Reject misaligned sign tensors (shader indexes data_s from offset 0) - Use the measured matcher distance as num_additional_fused_ops so the framework tracks the real matmul destination across empty reshape ops - Remove unconditional capability-probe stderr diagnostics - Shrink POW3 tables to the five reachable powers (n <= 4 everywhere) - Shorten comments to repo style (1-2 lines, ASCII) Verified on Radeon 890M: 7/7 generation checks, sustained decode unchanged.
|
Thanks for the careful review - all seven points addressed in 41c7d91:
Verified on Radeon 890M (Windows, MinGW Vulkan build): 7/7 generation checks, sustained decode speed unchanged. |
|
PR #187 validated on RDNA4 (gfx1201): ~4.75x decode, bit-identical Tested this PR on a Radeon AI PRO R9700 (Navi 48 / gfx1201, RADV) since the ResultBonsai 2 27B PTQ1_0, single stream, same machine/card/flags, only the build changed:
Decode measured 2x on the baseline and 4x on the PR, spread under 0.15 t/s. Output is bit-identical: 4/4 prompts at That is substantially better than the +76% reported on the 890M in #186. If that Measurement note that may save someone a bad numberMy first PR #187 reading was prefill 12.9 t/s — a 10x regression. It was a cold-start Context for scaleOn the same card and build, Qwen3.8-27B UD-Q5_K_XL (a 19.45 GiB conventional quant) SetupUnrelated confirmation of #204Loading Reproducible 3/3. Vulkan has no PQ2_0 pipelines ( |
bri-prism
left a comment
bri-prism
left a comment
There was a problem hiding this comment.
Agent review: posted by the maintainer's coding agent at their request.
Request changes: P2 - Validate every elided reshape consumer before fusing.
At ggml/src/ggml-vulkan/ggml-vulkan.cpp:10101-10107, the matcher checks only that MUL has one use. In MUL -> RESHAPE -> hinted MUL_MAT, that remains true when another operation also consumes RESHAPE, or the view is requested as an output. The fusion writes only the matmul destination and skips MUL, leaving that other consumer with an unmaterialized input.
A local GGML graph probe with an additional SCALE consuming RESHAPE confirms MUL uses=1 and the exact single-use helper returns true, while RESHAPE uses=2. The later fusion framework checks memory overlap, but does not reject this external consumer. Please validate every skipped intermediate/view and its output flag, or preserve the intermediate value.
The earlier pipeline-existence, sign-alignment and fusion-distance comments are addressed in this head. This new finding was validated at graph/use-count level; Vulkan numerical execution was not performed.
Reviewed commit: 41c7d913b8662cbd0378a4a00cf11a2008a06422.
|
4.75x on RDNA4 with bit-identical output - thank you for running this, that is a much stronger result than the 890M numbers and exactly the kind of cross-device data point the PR needed. The cold-start prefill artifact note is a good catch too - worth knowing for anyone benchmarking the first request after load. For anyone following along: the multiplier being larger on discrete RDNA4 than on the 890M iGPU makes sense given the decode kernel is ALU-bound before this change - the more ALU headroom the card has relative to memory bandwidth, the more the serial trit recurrence was costing. If others have RDNA2/3/4 or Intel/Arc data points, they would be welcome here as well. |
…mard inverse - Add mul_mat_vec_ptq1_0.comp: 16 threads per block with per-thread region assignment, one register multiplier per 8-element group. 2.30 -> 6.7 t/s decode on Radeon 890M (2.9x), bit-identical output. - mul_mm_funcs.glsl PTQ1_0 A-load: replace 16 ptq1_0_trit() calls per row with one vector decode using the group's shared exponent. Integer >>8 then float-convert (uint 0-1 wraps). pp512 83 -> 102 t/s (+23%). - qwen35.cpp MTP draft graph: apply the hadamard inverse after the MTP embedding lookup, mirroring build_inp_embd(). Without it, grafted-MTP GGUFs (token_embd in latent domain) fail context creation. Enables draft-mtp speculative decoding on ternary Bonsai: 0.47-0.82 acceptance with a Qwen3.8-27B head grafted by decent-jawfish/graft_mtp.py. Verified on Radeon 890M Vulkan (MinGW): 7/7 generation checks, correct spec-decode output at temperature 0.1.
|
Adding three more commits worth of work to this PR - all verified on the same Radeon 890M Vulkan setup: 1. Dedicated PTQ1_0 matvec kernel ( 2. Fast coopmat/mul_mm A-load for PTQ1_0 ( One trap worth flagging for anyone touching this: the natural one-liner 3. MTP hadamard inverse for grafted-head models ( With decent-jawfish's |
|
Vulkan follow-up data point for anyone watching this PR: the dedicated-kernel config from this branch (with the same n-max-1 MTP setup as #217/#218 use) was verified end-to-end on a second machine today - a Ryzen AI 9 HX 470 iGPU (Radeon 890M, RDNA 3.5, Vulkan/MinGW) running Windows - and the same algorithm family holds up there:
So the n-max 1 pick is not a Blackwell-specific artifact; it held on RDNA 3.5 Vulkan as well. |
Port of the algorithm ideas from PrismML-Eng#218 to the dedicated kernel: NUM_ROWS=4 per workgroup so every activation fetch is shared by four weight rows (cuts B-vector traffic 4x), plus SIMD-in-register trit decode on uvec4 lanes (branchless). Pipeline spec updated to 4 rows for the f32 matvec. Measured on Radeon 890M (RDNA 3.5, Vulkan/MinGW): PTQ1_0 tg128 5.80 -> 7.01 t/s in this tree (+21%), bit-identical decode, 3/3 generation checks pass. On the Sep-18-updated tree the same kernel measures 6.90 t/s. Co-authored-by: Hermes Agent <noreply@nousresearch.com>
|
Kernel v2 pushed in 42ee869 — porting the launch-geometry ideas from #218 into the dedicated Vulkan matvec:
Measured on Radeon 890M (RDNA 3.5, Windows, Vulkan/MinGW) in this PR tree: PTQ1_0 tg128 5.80 -> 7.01 t/s (+21%), bit-identical decode, 3/3 generation checks. (On a tree with Sep-18 upstream Vulkan improvements the same kernel measures 6.90.) Note the dp4a integer-dot trick that powers the CUDA version doesn't exist in Vulkan — this port captures the memory-traffic and decode-parallelism ideas instead, which is where the transferable win was. |
The GDN recurrence in hybrid models (qwen35 family) is bandwidth-bound on the recurrent state pool: every decode step reads and writes the full fp32 state across all recurrent layers. Halving that traffic to bf16 is the single biggest decode win left on bandwidth-starved iGPUs. - llama-model: opt-in env flags LLAMA_SSM_BF16_STATE (state pool s) and LLAMA_SSM_BF16_CONV (conv pool r) allocate the hybrid recurrent pools as BF16; default behavior unchanged - vulkan: new scale_bf16 pipeline (scale.comp with bf16 A/D types) plus supports_op / dispatch entries so GGML_OP_SCALE runs on BF16 tensors Measured, Radeon 890M (RDNA 3.5), Bonsai-2-27B Q2_0-fork + grafted MTP head, n-max 1: single-stream 10.4 -> 12.7 t/s (+22%), batch -np 16 18.46 -> 20.30 t/s aggregate. Quality: 5/5 short-form gates plus three 900-token generations with correct facts computed at the tail (bf16 state error does not compound into answers on this model). Co-authored-by: Hermes Agent <noreply@nousresearch.com>
|
bf16 SSM state pools shipped in 5360874 — the biggest decode win yet for hybrid (GDN) models on Vulkan: The recurrence is bandwidth-bound on its fp32 state pool — every decode step reads+writes the full state on every recurrent layer. Measured on Radeon 890M (RDNA 3.5), Bonsai-2-27B Q2_0-fork + grafted MTP n-max 1:
Default behavior is unchanged (opt-in env flags). This should transfer directly to every qwen35/hybrid model on any bandwidth-limited Vulkan device. |
Implements the rows-indexed GDN variant (src[6]) on Vulkan, previously CPU/Metal-only, plus an optional bf16 state pool: - gated_delta_net.comp: USE_STATE_ROWS variant reads the initial state directly from the cache row given by an int32 index buffer (binding 7) instead of a per-layer gather; STATE_BF16 variant reads a bf16 state pool with the bit-exact uint16<<16 conversion - 6 new pipelines (rows f32 / rows bf16state x 3 reduce modes x kda) - dispatch + supports_op accept src[6] with F32 or BF16 state - ggml.c: rows op allows BF16 state tensors - qwen35: Vulkan added to the rows-mode device allowlist Measured, Radeon 890M, Bonsai-2-27B Q2_0-fork + MTP n-max 1, together with LLAMA_SSM_BF16_STATE=1 from the previous commit: 10.4 -> 12.7 t/s single-stream vs stock path, 5/5 generation gates. Eliminates the per-layer GET_ROWS/CPY state bookkeeping (~192 launches/token). Co-authored-by: Hermes Agent <noreply@nousresearch.com>
|
Vulkan gated_delta_net rows-mode + bf16 state — commit 9606246. This closes the gap the code itself acknowledged: "rows mode is implemented on CPU and Metal only; other GPU backends reject it in supports_op." Vulkan now implements it:
Every qwen35/hybrid-GDN model on any Vulkan device benefits; Metal users keep their existing path unchanged. |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
It contains compile-blocking graph references, incorrect BF16 handling, an out-of-bounds matvec read, and incomplete FWHT fusion plumbing.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 4
Open (5)
Resolved since last review (7)
MUL_MATis normally two graph positions away (MUL, emptyRESHAPE,MUL_MAT), but…signsis bound withallow_misalign=true, which rounds its descriptor offset down, but the…nas at most 4 for…nis bounded by 4, not 119. Keeping 64 powers here duplicates 59…
| const uint y_idx = i * QUANT_K + e0; | ||
|
|
||
| // B vectors fetched ONCE for all 4 rows | ||
| vec4 bv[NUM_COLS][2]; | ||
| [[unroll]] for (uint j = 0; j < NUM_COLS; ++j) { | ||
| bv[j][0] = vec4(data_b_v4[(j*p.batch_stride_b + b_offset + y_idx) / 4]); | ||
| bv[j][1] = vec4(data_b_v4[(j*p.batch_stride_b + b_offset + y_idx) / 4 + 1]); | ||
| } | ||
|
|
||
| // weights decoded once per (row, group) | ||
| vec4 wf[4][2]; | ||
|
|
||
| [[unroll]] for (uint n = 0; n < 4; ++n) { | ||
| if (n >= num_rows) { break; } | ||
| const uint ib0 = a_offset + (first_row+n)*num_blocks_per_row; | ||
| if (i >= num_blocks_per_row) { continue; } |
| string_to_spv("repeat_i16", "repeat.comp", {{"A_TYPE", "int16_t"}, {"D_TYPE", "int16_t"}}); | ||
|
|
||
| string_to_spv("scale_f32", "scale.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}}); | ||
| string_to_spv("scale_bf16", "scale.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"FLOAT_TYPE", "float"}}); |
| GGML_ASSERT(g->type == GGML_TYPE_F32); | ||
| GGML_ASSERT(beta->type == GGML_TYPE_F32); | ||
| GGML_ASSERT(states->type == GGML_TYPE_F32); | ||
| GGML_ASSERT(states->type == GGML_TYPE_F32 || states->type == GGML_TYPE_BF16); |
| // multi-layer hidden-state tap: collect the captured layer outputs here in | ||
| // capture order, then concatenate them along dim0 after the layer loop. | ||
| std::vector<ggml_tensor *> h_capture(cparams.n_capture_layers, nullptr); |
| // FADI-FUSION: sign vector [N*n_blk] for the fused MUL(signs)+FWHT variant only | ||
| #ifdef FWHT_SIGNED | ||
| layout(binding = 2, std430) readonly buffer S { float data_s[]; }; |
…Q1_0 tail guard before B load
|
Review responses pushed in 157b728:
On the bf16 scale note: agreed the committed FWHT_SIGNED: acknowledged — the signed variant is dead code as committed; either the plumbing lands complete or the branch gets compiled out. Tracked for the follow-up commit. |
Data point: Intel Arc B390 (Panther Lake Xe3 iGPU), Vulkan / Windows — 3.7x decode confirmed, plus one reproducible
|
| build | tg128 (r1 / r2) | pp512 (r1 / r2) |
|---|---|---|
| prism | 1.59 / 1.57 | 206.6 / 169.9 |
| #187 | 5.87 / 5.90 | 225.3 / 211.5 |
3.73x decode, stable to ±0.01 within each run. Consistent with the 2.9x on the 890M and 4.75x on RDNA4 — this part lands between them.
(For prefill I would not claim anything: pp512 on this box swings ~20% round to round from page-cache state, visible in the two prism rows.)
Regression: one extra GATED_DELTA_NET failure
test-backend-ops test -o GATED_DELTA_NET -b Vulkan0, failing-case counts, same machine, back to back:
| run | prism 422590f5d |
#187 157b7289e |
|---|---|---|
| full suite | 3 | 4 |
| 1 | 3 | 4 |
| 2 | 2 | 4 |
| 3 | 3 | 4 |
The data for this op is unseeded, so prism's own count moves between 2 and 3 — but it never reached 4, and #187 was 4 in 4/4 runs. The extra case is:
GATED_DELTA_NET(type=f32,head_count=4,head_size=128,n_seq_tokens=1,n_seqs=2,
v_repeat=1,permuted=0,kda=0,K=1,rows_mode=0,cache_rows=-1,raw_gates=1)
ERR ~0.92 against the CPU reference, i.e. the output is essentially uncorrelated, not a tolerance issue. The other three (n_seqs=1/v_repeat=1, n_seqs=2/v_repeat=2, n_seq_tokens=64) fail identically on stock prism and are pre-existing.
This PR does touch gated_delta_net.comp, so that is the obvious place to look — though note the failing case reports rows_mode=0, so the new USE_STATE_ROWS path may not be what is doing it, and I have not bisected it. Flagging the measurement rather than a diagnosis.
Two housekeeping things
The diff is ~99.8% line-ending churn. src/models/qwen35.cpp, vendor/cpp-httplib/httplib.cpp and httplib.h are converted to CRLF, which is what produces the +22923/-22591. Ignoring CR the real change is:
src/models/qwen35.cpp 18+ 1-
vendor/cpp-httplib/httplib.cpp 5+ 2-
vendor/cpp-httplib/httplib.h 3+ 2-
The cpp-httplib edit is not mentioned in the description. It swaps CreateFile2 for CreateFileW and replaces the #error with #define _WIN32_WINNT 0x0A00 — a MinGW build workaround in a vendored dependency. Worth splitting out (or sending upstream), since it will silently revert on the next vendor bump.
Also, the qwen35.cpp hunk is the same MTP Hadamard inverse as #205, which is already merged and is the current prism HEAD — so that part may just drop out on a rebase.
|
Correction to my earlier comment: the fourth I said this PR "adds a fourth failing case" and described it as "a distinct regression". Having gone back and compared the full case strings rather than the counts, that is not what is happening. The case is: Note This PR implements that path ( So the correct reading is: this PR legitimately enables a shape, and that shape immediately lands on a pre-existing bug. I filed the underlying bug separately as #237 before I understood the connection — the Vulkan shader applies I have a fix for the root cause up as #239 (declines Apologies for the mischaracterisation; "regression" was the wrong word and I should have diffed the case strings before using it. The other points in my earlier comment — the 3.73x decode measurement, the CRLF line-ending churn, and the undisclosed |
|
Would you be up for rebasing this branch once #239 lands? To be clear about what it is and isn't for: nothing in your code needs fixing for this. As above, the fourth No urgency — #239 is unmerged, so the natural order is to let it land on One thing worth sorting out before you do, though, or the rebase will be unpleasant: this branch is now 29 commits behind almost all of which is CRLF conversion rather than real change. Ignoring line endings, the actual diff is If you normalise the line endings first — reverting those three files to LF and reapplying just the real hunks — the rebase becomes almost trivial, and the PR also becomes reviewable, since right now the Vulkan shader work that is the point of this PR is buried under the churn. The Happy to re-run the full |
…27B (build config, launch flags, env vars, measured ladder, pitfalls)
github-actions
Bot
added
the
documentation
|
Full speed recipe published in this PR: docs/vulkan-890m-bonsai2-recipe.md Results on Radeon 890M (Ryzen AI 9 HX 470): Bonsai 2 27B at 1.85 t/s stock to 11.5-12 t/s over HTTP (14.1 peak server-side) with MTP spec-decode, acceptance 0.84, 5/5 quality gate. Includes: complete build config, launch flags, env vars, measured ladder for every optimization, and measured dead ends (n=2 spec, KV-q8, leaner quants). Key finding for Vulkan maintainers: the raw-gates Vulkan exclusion in qwen35.cpp is correct - verified corrupted output on all shader variants; iGPU decode is dequant-bound before bandwidth-bound (leaner quants measured slower). Hope it helps other iGPU users. |
a093f16 to
c6fab03
Compare
bri-prism
left a comment
bri-prism
left a comment
There was a problem hiding this comment.
Re-reviewed at c6fab03. My original finding is resolved. The PR has grown a lot since I looked, and most of what I have now is about that rather than about the kernels.
P2 is resolved, by removal rather than repair. At 41c7d91 the matcher checked ggml_node_has_n_uses(cgraph, node_idx, 1) on the MUL and then accepted MUL -> RESHAPE -> hinted MUL_MAT, which stayed true when something else also consumed the RESHAPE. At head that matcher is gone, ggml_vk_can_use_fwht returns false outright when ctx->num_additional_fused_ops != 0, and the only MUL fusions left in the file are the pre-existing RMS_NORM+MUL+ROPE and snake patterns. There is no longer a path that elides an intermediate without materializing it, so the failure I described cannot occur.
Worth noting the title and description still advertise "add MUL+FWHT fusion". Since the fusion is no longer in the change, both should be updated so a reader is not looking for code that was withdrawn.
The scope has roughly quadrupled and I think it should be split. What I reviewed was PTQ1_0 decode plus a fusion. What is here now is fifteen files spanning a dedicated PTQ1_0 matvec kernel, a coopmat A-load path, a 4-row workgroup rewrite, bf16 SSM state pools, a gated_delta_net rows-mode port, an MTP hadamard graft, a recipe document, and edits to vendored third-party code. Several of those are individually worth landing. Bundled, they are hard to review, hard to bisect and hard to revert.
The vendored httplib edits should not be in this PR at all. vendor/cpp-httplib/httplib.h replaces upstream's deliberate
#error "cpp-httplib doesn't support Windows 8 or lower. Please use Windows 10 or later."with #define _WIN32_WINNT 0x0A00, which silences the guard by asserting the answer rather than fixing the build configuration that tripped it, and does so inside a dependency we do not own. httplib.cpp separately swaps CreateFile2 for CreateFileW. Both are Windows build fixes with no relationship to Vulkan PTQ1_0 decode, both will be silently reverted the next time the vendor directory is refreshed, and the _WIN32_WINNT one changes behaviour for every consumer of that header. If the build genuinely needs these, they belong in a separate PR where they can be discussed on their own terms, and the _WIN32_WINNT definition belongs in the build configuration rather than in vendored source.
The headline number needs attribution before it means anything. The recipe claims 1.85 to 14 tokens per second, which is roughly 7.6x. That figure currently sits on top of at least five independent changes: the matvec kernel, the coopmat A-load, the 4-row workgroup rewrite, the bf16 state pools, and the GDN rows-mode port. One commit message separately claims +22% for the state pools alone.
An aggregate speedup over a bundle does not tell you which parts earned it, and on this project we have landed bundles before where one component was neutral or negative and the aggregate hid it. Could you give a per-change ladder, each step measured with the previous ones in place, so the marginal contribution of each is visible? That also tells you whether anything in the bundle can be dropped, which is the cheapest way to shrink a PR this size.
What I have not done. I have not executed anything on Vulkan, have not verified the numerical correctness of the new matvec kernel or the bf16 state pools, and have not reviewed the GDN rows-mode port or the MTP graft in any depth. Those need a reviewer with the hardware, and given the 890M recipe is the validation path here, ideally on that device.
I am leaving my change request in place for now, on the vendored dependency edits rather than on anything in the kernel work. Split those two files out and I will clear it.
|
Split per review feedback — rebased on current prism:
Dropped from #187: httplib vendored edits (removed entirely) and MUL+FWHT fusion (withdrawn earlier). Notes:
Both split PRs measured on Radeon 890M (RDNA 3.5) with Bonsai-2-27B: details in the PR descriptions. |
What
Three Vulkan-side optimizations for PTQ1_0 decode on ternary (Bonsai 2) models, plus an MTP graph fix.
1. Trit table for the serial trit recurrence (
ptq1_0.glsl+dequant_funcs.glsl)3^n mod 256 has period 64, so the serial remainder recurrence collapses to one lookup. Bit-identical output.
2. Vectorized
dequantize4for PTQ1_0Four-way float decode replacing per-element scalar work.
3. MUL+FWHT fusion (
fwht.comp,ggml-vulkan.cpp)Folds the sign-vector MUL that precedes the FWHT-hinted MUL_MAT into the matvec kernel, with a guarded matcher (signed pipeline must exist, sign tensor aligned, measured fusion distance).
4. Dedicated PTQ1_0 matvec kernel (
mul_mat_vec_ptq1_0.comp) - added in 4b8092cPer-thread region assignment, register multiplier per 8-element group. 2.30 -> 6.7 t/s decode (2.9x) on Radeon 890M, bit-identical.
5. Fast coopmat A-load for PTQ1_0 - added in 4b8092c
Group-shared exponent vector decode replacing 16
ptq1_0_trit()calls. pp512 83 -> 102 t/s (+23%).6. MTP hadamard inverse (
qwen35.cpp) - added in 4b8092cMTP draft graph skipped the hadamard inverse on its embedding lookup; grafted-head GGUFs failed context creation. Enables
--spec-type draft-mtpon ternary Bonsai (acceptance 0.47-0.82 with a grafted Qwen3.8 head, +13-26% single-stream with the head on CPU).Measurements
Radeon 890M (iGPU, RDNA 3.5), Windows, MinGW Vulkan build, Bonsai 2 27B:
Cross-device: RDNA4 (gfx1201, RADV) reports 4.75x decode, bit-identical (zamorake, validated independently).
Verification
Assisted by AI (Hermes agent); verified by hand on hardware.