Repository navigation
metal: read FP16 and BF16 affine constants directly for FP32 inputs - #19
Merged
Merged
Conversation
quantized_matmul used to cast an FP32 input's FP16 or BF16 scales and biases to FP32 before every call: two extra kernels per projection, or a second FP32 copy of every constant kept resident. On Metal the op now keeps them narrow. The one-row qmv_fast kernel reads them as they are and widens each value in registers (a new affine_qmv_fast_mixed entry). Every other route widens them inside eval_gpu first, as the op did. Outputs are bitwise identical to widening first; other backends and CPU streams keep the old cast.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With an FP32 activation and FP16 or BF16 scales and biases,
quantized_matmulwidens the constants on every call. This keeps them narrow on Metal: a newaffine_qmv_fast_mixedkernel reads them in registers for the one-row path, and every other route widens them insideeval_gpu. Other backends and CPU streams are unchanged.M5 Pro, Release, inference mode (
train(false)asserted), plain greedy decode of a 2-bit signed-Hadamard target, 128 tokens, two alternating reps, tokens identical in every arm:That is -7.6 percent per token and -1.49 GiB. DFlash2 does not regress (coding 13.13 vs 13.23 ms/tok, essay 69.6 vs 69.8).
Correctness: 193 of 193 cases are bitwise equal to widening first (FP16 and BF16 constants, 2, 3, 4 and 8 bits, M 1 to 64, several K and N including unaligned N, plus a CPU stream).