Feat/native gguf llama - #327
Open
amodhyh wants to merge 9 commits into
Open
Conversation
Adds parse_gguf_config to map GGUF metadata to ModelConfig, iter_gguf_weights to stream and fuse QKV/GateUp tensors, and convert_llama_to_gguf to swap dense PyTorch layers with custom GGUF GPU kernels.
- Clamp RoPE max_position to 8192 to prevent massive pre-allocation OOMs - Enable MoE offload routing by setting `moe_enabled` based on expert count - Filter MoE expert tensors in `iter_gguf_weights` using include_moe_experts flags - Implement shared expert mapping and fused gate/up aggregation - Prevent 720 MiB GPU OOMs during dense pass by skipping expert allocations
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request adds comprehensive support for loading and running Llama 4 (and Llama) models in GGUF format within the FreeToken codebase. The changes introduce GGUF-specific model configuration and weight loading for Llama, update tokenizer handling for new special tokens, and extend the model registry and architecture mappings to recognize and instantiate Llama GGUF models. Additionally, it updates the Llama model implementation to support new routing logic for MoE and shared expert layers.
Llama GGUF Model Support
gguf.pyadapter infreetoken.models.llamafor parsing GGUF configs and iterating weights, including support for MoE and shared expert routing. Also includes a conversion function to adapt Llama models to GGUF format.model.py) to support GGUF models: conditional initialization and conversion, new MoE/shared expert routing logic in decoder layers, and correct handling of GGUF-specific weight formats. [1] [2] [3] [4]Tokenizer and Special Token Handling
<|eot|>and<|eot_id|>as end-of-turn tokens, ensuring correct EOS token selection and stop id computation. [1] [2] [3]Configuration and Registry Updates
llamaandllama4, mapping them to the correct GGUF model class.LlamaGGUFForCausalLMin the model registry, specifying the GGUF-specific config and weight loading functions.__all__for import convenience.References #34. This PR introduces the LLaMA implementation for the overarching feature request.