Transformers Adds Direct Support for llama.cpp GGUF Quants

Transformers Adds Direct Support for llama.cpp GGUF Quants Hugging Face has integrated direct GGUF model execution into transformers, allowing users to run quantized checkpoints locally on Apple Silicon using standard transformers APIs while reusing ggml's Metal kernels. transformers can load GGUF checkpoints directly via from_pretrained and host them through an OpenAI-compatible serving endpoint. The implementation reuses ggml Metal layer kernels via the kernels library to achieve inference throughput close to native llama.cpp. Engineers reduced Python generation overhead by removing unnecessary padding masks early and asynchronously deferring stopping checks. The initial packed execution path is restricted to Apple Silicon (MPS) and currently supports Qwen3.5 dense and MoE architectures, along with compatible Qwen3.8 checkpoints. Loading and Serving GGUF Checkpoints Users on Apple Silicon can load GGUF models directly by specifying the repository ID and file in from_pretrained without extra configuration. When compatible ggml Metal kernels are detected, the weights stay packed on Metal rather than dequantizing. Checkpoints can also be deployed locally using transformers serve to expose an OpenAI-compatible endpoint for desktop clients such as Jan or Pi, with configurable flags for handling thinking templates. Performance and Complementary Tooling Tested on an M2 Max MacBook Pro against llama-bench, transformers achieved token generation throughput close to llama.cpp across dense and MoE models. While llama.cpp remains the recommended runtime when standalone local execution efficiency is the main priority, the transformers integration allows developers to inspect activations with hooks, test new decoding logic, validate GGUF conversions against original weights, and dequantize models for fine-tuning. Kernel Acceleration and Loop Optimizations To keep eager execution fast in Python without relying on torch.compile, transformers pulls specialized Metal kernels from ggml via the kernels library, including fused normalization, flash attention, gated delta nets, and a custom top-k routing kernel. Additionally, the generate loop itself was updated across all transformers models to drop unnecessary attention masks for unpadded decoder inputs and defer the stopping evaluation asynchronously, avoiding idle CPU waits during GPU execution. Current Scope and Limitations Packed GGUF inference is currently limited to single interactive conversations on Apple Silicon via MPS. Unsupported environments fall back to dequantizing the model into memory. Batching and padded inputs do not yet leverage the mask optimizations, and architectural coverage is presently restricted to Qwen3.5 and compatible Qwen3.8 models, with broader model and batch support planned for future updates.

Transformers 支持直接运行 GGUF 模型，借助 ggml 内核加速本地推理 Hugging Face 宣布在 Transformers 库中支持高效运行 GGUF 量化模型。通过复用 llama.cpp 底层的 ggml Metal 内核并优化生成循环，用户无需脱离熟悉的原生 API 和 PyTorch 生态，即可在 Apple Silicon 设备上实现接近 llama.cpp 的推理性能。 用户可通过 from_pretrained 传入 gguf_file 参数直接加载 Hub 上的 GGUF 检查点，无需额外配置即可调用 Metal 专用算子。 支持通过 transformers serve 快速部署兼容 OpenAI 格式的本地推理服务，并可对接 Jan 或 Pi 等第三方客户端。 集成通过 kernels 库复用 ggml 的 Metal 内核（如量化计算、归一化、注意力算子），并结合自研 MoE top-k 算子减少 GPU 计算开销。 优化了 generate 生成循环，通过提前剥除无用注意力掩码并异步延迟停止检查，减少了 CPU 与 GPU 之间的同步阻塞。 GGUF 加载与服务化部署 用户安装最新版 Transformers 和 kernels 库后，可在 AutoModelForCausalLM 和 AutoTokenizer 的 from_pretrained 中直接传入 GGUF 文件名。系统会自动启用 Metal 打包权重和 ggml-attn 注意力实现；若环境不兼容则会回退至反量化或 SDPA 模式。此外，利用 transformers serve 命令即可快速启动与 OpenAI 接口规范兼容的本地 API 服务，方便接入各类前端客户端。 内核复用与生成循环加速 为在 Python/PyTorch 环境下逼近 llama.cpp 的吞吐表现，该实现复用了 ggml 的 Metal 底层内核（包括量化矩阵乘法、归一化、flash attention 等），避免每次解码展开全量权重。同时，Transformers 改进了 generate 逻辑：对无 padding 的单序列提前移除全 1 掩码，并将停止条件检测移至异步流水线，减少 CPU 等待 GPU 完成计算的同步停顿，显著提高了端到端生成速率。 适用场景与当前局限 对于单纯的本地运行场景，llama.cpp 依然是官方推荐的独立引擎；但 Transformers 的集成使得在 PyTorch 中检查激活值、验证量化误差、尝试自定义解码算法或直接基于 GGUF 微调变得更加便捷。目前打包内核推理仅支持 Apple Silicon MPS 后端，架构覆盖仅限 Qwen3.5 等少数系列，带 padding 的批量推理也尚待进一步优化。

We're adding support for running GGUF models efficiently in transformers , so you can use checkpoints sized for your laptop's memory through the familiar transformers APIs. Pick a GGUF from the Hub, load it with from_pretrained , and start generating on your own machine. Running AI models on your laptop has become much easier, and llama.cpp has been a big part of that. Its inference engine powers local AI tools such as Ollama, LM Studio, and Jan. Alongside projects like MLX , it has helped make local inference a practical option for everyday use. A recent example of what local AI can feel like: This is where we are right now. And i’m not gonna lie it feels pretty magical 🧙‍♀️ Qwen3.6 27B running inside of Pi coding agent via Llama.cpp on the MacBook Pro For non-trivial tasks on the @huggingface codebases, this feels very, very close to hitting the latest Opus in Claude… pic.twitter.com/lsIxLoUneU — Julien Chaumond (@julien_c) April 24, 2026 GGUF , developed by the llama.cpp team, is a widely used format for local inference. The team also shares quantized checkpoints under ggml-org on the Hub . Publishers such as Unsloth , LM Studio Community , and bartowski also provide ready-to-use GGUF checkpoints in a range of quantizations, so users can pick the version that fits their machine. GGUF models have been downloaded millions of times. We want to make it easier to run these models locally with transformers, too. Compatibility is only useful if the model is pleasant to run. To bring performance close to llama.cpp, we're reusing its underlying ggml kernels through the kernels library, and reducing overhead in generate . Our initial focus is local inference on Apple Silicon, starting with the Qwen3.5 architecture. What is the GGUF file format? GGUF packages model weights and metadata, including tokenizer information and an optional chat template, in one file. It supports different quantization levels, letting you trade some precision for a smaller memory footprint. Variants such as Q4_K_M mix tensor precisions, using mostly 4-bit weights while keeping sensitive tensors at higher precision. Here's how quantization changes the file size of Unsloth's Qwen3.5-4B : GGUF variant File size Tradeoff BF16 8.42 GB Unquantized reference Q6_K 3.53 GB More precision than the smaller variants Q5_K_M 3.14 GB A middle ground between size and precision Q4_K_M 2.74 GB A practical starting point for local inference We suggest starting with Q4_K_M , then trying Q5_K_M or Q6_K if you have more memory available. More aggressive quantization can help larger models fit, but the quality tradeoff depends on the model and the task. Evaluate it on the work you actually want the model to do. The Hub's GGUF documentation describes the available quantization types. Load GGUF with transformers To get started, you need: An Apple Silicon Mac . A PyTorch version supported by the published ggml-quantization kernel builds , usually the two latest PyTorch releases. The latest version of transformers (main for now, until the next release) and a compatible version of kernels . pip install -U "git+https://github.com/huggingface/transformers.git" kernels To load a GGUF model, pass its Hub model_id and filename as gguf_file to from_pretrained . No extra configuration is needed: when the weights stay packed on Metal, transformers automatically loads the compatible ggml/Metal layer kernels and uses ggml-org/ggml-attn as the attention implementation. If that kernel cannot be fetched, the model falls back to "sdpa" with a warning, and you can always force "sdpa" by passing attn_implementation="sdpa" explicitly. See the GGUF documentation for more loading options. import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "unsloth/Qwen3.5-4B-GGUF" filename = "Qwen3.5-4B-Q4_K_M.gguf" tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename) model = AutoModelForCausalLM.from_pretrained( model_id, gguf_file=filename ) That is the only GGUF-specific step. Everything after it is the standard transformers API: messages = [{ "role" : "user" , "content" : "Explain why the sky is blue in a few sentences." }] inputs = tokenizer.apply_chat_template( messages, tokenize= True , add_generation_prompt= True , return_dict= True , return_tensors= "pt" , ).to(model.device) with torch.inference_mode(): outputs = model.generate(**inputs, max_new_tokens= 256 ) print (tokenizer.decode(outputs[ 0 ], skip_special_tokens= True )) Without a compatible quantization kernel, the loader falls back to dequantizing the model and uses more memory. Serve GGUF with your preferred interface You can also use the same checkpoint with transformers serve , which exposes an OpenAI-compatible API: pip install -U "transformers[serving] @ git+https://github.com/huggingface/transformers.git" kernels transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf" The model argument uses <model_id>:<filename>.gguf : before the colon is the Hub repository ( unsloth/Qwen3.5-4B-GGUF ), and after it is the file to load ( Qwen3.5-4B-Q4_K_M.gguf ). This selects a specific quantization from a repository that may contain several. For models whose chat template supports thinking, add --reasoning off to skip it or --reasoning on to enable it. The default, --reasoning auto , follows the chat template’s default. See the reasoning options for details. You can connect a client such as Jan or Pi by adding a custom OpenAI-compatible provider with these settings: Setting Value Base URL http://localhost:8000/v1 Model ID unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf transformers runs the model on your Mac, while the client provides the conversation interface. The same endpoint can be used by other clients that support this API. Benchmarking against llama.cpp Our reference for local inference performance is llama.cpp. The comparison below focuses on three GGUF checkpoints: a small dense model, a larger dense model, and a mixture-of-experts model. The llama.cpp column comes from the llama-bench tool (build 5f55650a7 , release b10200, Metal backend from ggml 0.18.0), run as llama-bench -m <file> -p 0 -n 128 -r 3 , which reports tg128 : the token-generation rate over 128 decoded tokens, averaged across three repetitions, with prompt processing excluded. The transformers column is generate producing the same 128 tokens from a 12-token prompt, best of three warmed runs, and it includes prefill. Measured on a MacBook Pro M2 Max, 32 GB unified memory, macOS 26.6, PyTorch 2.12.1, kernels 0.17.0, plugged in. The benchmark script import time import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id, filename = "unsloth/Qwen3.5-4B-GGUF" , "Qwen3.5-4B-Q4_K_M.gguf" model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename) tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename) inputs = tokenizer( "The capital of France is Paris. The capital of Germany is" , return_tensors= "pt" ) inputs = inputs.to(model.device) with torch.inference_mode(): model.generate(**inputs, max_new_tokens= 8 , min_new_tokens= 8 , do_sample= False ) # warm up torch.mps.synchronize() for _ in range ( 3 ): time.sleep( 90 ) # let the machine cool: back-to-back runs decay by 10% or more start = time.perf_counter() model.generate(**inputs, max_new_tokens= 128 , min_new_tokens= 128 , do_sample= False ) torch.mps.synchronize() print ( f" { 128 / (time.perf_counter() - start): .1 f} tok/s" ) For the other column: llama-bench -hf unsloth/Qwen3.5-4B-GGUF:Q4_K_M -p 0 -n 128 -r 3 Transformers is close to llama.cpp across all three checkpoints. The chart uses the same measurements described above; it does not imply identical benchmark conditions, since the Transformers measurement includes prefill while llama-bench reports decode-only throughput. transformers and llama.cpp When GGML and llama.cpp joined Hugging Face , we described their complementary roles: llama.cpp provides a foundation for local inference, while transformers provides a foundation for model definition. GGUF support brings those two closer together. llama.cpp remains our recommended engine when your priority is efficient local inference. Its dedicated runtime, memory management, and broad hardware support are built around that goal. This integration gives developers a convenient way to work with the same GGUF checkpoints inside transformers: Experiment with GGUF in Python and PyTorch. Inspect intermediate activations with hooks, modify a model's forward pass, or prototype custom layers using familiar PyTorch tools. Evaluate GGUF models. Use your existing transformers evaluation workflows to measure the quality of quantized checkpoints. Validate GGUF conversions. For us as developers, loading the original checkpoint and its GGUF conversion in transformers makes it easier to check that the weights were converted correctly, accounting for quantization error. Try new decoding ideas. Use custom logits processors and stopping criteria with generate , or write your own generation loop in Python. Fine-tune from a GGUF checkpoint. Dequantize the weights and continue with a standard transformers training workflow. For that last case, use GgufConfig(dequantize=True) : import torch from transformers import AutoModelForCausalLM, GgufConfig model = AutoModelForCausalLM.from_pretrained( "unsloth/Qwen3.5-4B-GGUF" , gguf_file= "Qwen3.5-4B-Q4_K_M.gguf" , quantization_config=GgufConfig(dequantize= True ), dtype=torch.bfloat16, ) Beyond GGUF: ggml kernels for more models The bigger opportunity is bringing ggml's performance to models that llama.cpp does not support. transformers already provides the PyTorch implementations of these architectures. With ggml kernels and quantization schemes available in PyTorch, we can work toward accelerating their supported operations without first implementing the entire model in llama.cpp. This is especially useful for new architectures, research models, and custom variants that may never receive a dedicated llama.cpp implementation. That opportunity extends beyond the GGUF format itself. A kernel operates on tensors; it does not require the whole model to come from a GGUF file. The same building blocks can be integrated into other transformers models and loading workflows. This also opens a path to other modalities: computer vision models, audio models, and multimodal models could reuse compatible attention, normalization, and matrix multiplication kernels without first having a full implementation in llama.cpp. Each architecture still needs integration and validation; the initial GGUF examples here cover text generation. Fast local inference with Python and PyTorch We also wanted to show how far we can get while keeping the model and generation loop in Python. With the right kernels and an efficient generation loop, Python and PyTorch can deliver strong local inference performance. The kernels handle the heavy computation, while the generation loop keeps the GPU busy by avoiding unnecessary synchronization. Our focus was to make eager execution fast without requiring torch.compile . For interactive use, we wanted a quick start and a steady stream of tokens, without compilation pauses or recompilation when input shapes change. The two main pieces of that work are the kernels and generate itself. Reusing ggml's Metal kernels A kernel is a small program that performs an operation on the GPU. PyTorch supplies general-purpose implementations; a specialized kernel can do less work, combine several operations, or read quantized weights directly in their stored format. The kernels library lets us distribute compatible builds of ggml's Metal kernels on the Hub and call them from transformers. That brings ggml's work into the PyTorch model without replacing the model with a separate inference runtime. Kernel What it does ggml-quantization Reads packed quantized weights for matrix operations, including the selected experts in an MoE model. It avoids expanding the whole weight matrix before each decode operation. ggml-norm Fuses normalization operations, including the zero-centered RMSNorm used by Qwen3.5 and Qwen3.8. ggml-attn Provides ggml's Metal flash attention for prompt processing and token decoding. ggml-gated-delta-net Accelerates the gated delta network used in the linear-attention layers of the Qwen3.5 and Qwen3.8 hybrid architectures. topk Selects the experts for each token in an MoE model, combining softmax and top-k routing. This is our own Metal implementation. The first four packages build on ggml's kernels; the top-k kernel addresses a separate bottleneck in MoE routing. Together they reduce the GPU work needed for each generated token. To show the contribution of the layer kernels, we compare the same packed GGUF checkpoints with and without them. The quantization kernel stays enabled in both configurations: disabling it would also change how weights are represented and would measure a different tradeoff. Keeping the CPU and GPU working together Faster kernels only help if the GPU has work to do. During generation, the CPU schedules GPU operations and controls the loop that produces the next token. Reading a result back from the GPU can force the CPU to wait until queued operations finish. Repeating even a small wait for every token can noticeably reduce throughput. Two changes address this in generate , which results in improvements for all transformers models (not just when running GGUF files): Drop an unnecessary attention mask early (#48814) . When a supported decoder-only input has no padding, its all-ones padding mask can be removed at the start of generation. Downstream attention code no longer needs to inspect that mask repeatedly to determine whether it can be skipped. Causal attention is still preserved. Defer the stopping check (#47975) . On supported paths, generate copies the stopping decision asynchronously and consumes it on the following step. The CPU can keep scheduling work while the GPU runs. Streaming tokens use the same approach, and any extra step past the stopping condition is removed from the result. These changes improve the generation loop around the model, so their usefulness extends beyond GGUF. They complement the kernel work: kernels reduce the cost of an operation, while fewer synchronization points let CPU scheduling and GPU execution overlap. These measurements keep all layer kernels enabled; the bars isolate the changes to the generation loop. Current limitations and next steps The initial target is a single interactive conversation on Apple Silicon. There are a few boundaries to keep in mind: The packed inference path is MPS-only for now. GGUF import through dequantization remains a separate option; support for the file format does not imply that packed kernels are available on every device. Padding and batching still need work. Unpadded inputs benefit from the mask optimization described above. Padded batches cannot take the same shortcut and can have lower performance. We want to extend the work to generate_batch on MPS. Architecture coverage is limited. The packed loader currently covers the Qwen3.5 dense and MoE architectures, including compatible Qwen3.8 checkpoints. Adding support for other architectures is relatively straightforward, and we’ll expand coverage gradually. If you have a GGUF model you would like to use in transformers, open an issue with the checkpoint and your use case. That will help us prioritize support for the models people are running locally. Acknowledgments We would like to thank Arthur Zucker for initiating this work and reviewing all of my PRs, and Cyril Vallez for the generate PRs. We are grateful to Sayak Paul , the llama.cpp team , and Bertrand Chevalier for their help integrating the kernels. We also thank Aritra Roy Gosthipaty and Pedro Cuenca for reviewing this blog post, and Lysandre Debut for overseeing the project.