Transformers now loads llama.cpp GGUF quants on Apple Silicon

Hugging Face published GGUF support in transformers on September 22, 2026, and the practical effect is narrow but real. A quantized checkpoint sized for your laptop now loads through from_pretrained and generates text from there. No conversion step, and no leaving PyTorch. You pass the Hub repository and the file name as gguf_file, and nothing else in the usual code changes. The weights stay packed, and the library fetches ggml's Metal kernels to read them in that form.

What you need, and what fits

The requirements are specific: an Apple Silicon Mac, a PyTorch version covered by the published kernel builds, transformers from the main branch until the next release, and a compatible kernels. If the attention kernel cannot be fetched, the model falls back to sdpa with a warning rather than failing, and without a compatible quantization kernel the loader dequantizes and uses more memory.

File size is the number that decides what runs on a given machine, and Hugging Face publishes it for Unsloth's Qwen3.5-4B. The same weights range from 8.42 GB unquantized to 2.74 GB at Q4_K_M, which is the variant the post recommends starting from. Q5_K_M and Q6_K are the steps up when memory allows.

• BF16: 8.42 GB, the unquantized reference.
• Q6_K: 3.53 GB, more precision than the smaller variants.
• Q5_K_M: 3.14 GB, between size and precision.
• Q4_K_M: 2.74 GB, the recommended starting point.

How close it gets to llama.cpp

Hugging Face measured against llama.cpp on a MacBook Pro with an M2 Max and 32 GB of unified memory, on macOS 26.6, with PyTorch 2.12.1 and kernels 0.17.0, plugged in. The llama.cpp side comes from llama-bench reporting decode-only throughput over 128 tokens, averaged over three runs. The transformers side generates the same 128 tokens from a 12-token prompt, best of three warmed runs, and includes prefill.

The post says transformers is close to llama.cpp across all three checkpoints it tested, and says plainly that the two measurements do not run under identical conditions. The per-checkpoint figures appear only in a chart, not in the text, so there is no number here to quote. Hugging Face also keeps its recommendation intact: llama.cpp remains its engine of choice when the priority is efficient local inference.

Why it is worth having anyway

The value is not a faster chatbot. It is that a GGUF checkpoint becomes an ordinary PyTorch model. You can hook into activations, change a forward pass, or run your existing evaluation workflow against the quantized file. You can also check a conversion by loading the original and the GGUF side by side. Fine-tuning from a GGUF checkpoint works through GgufConfig(dequantize=True).

The limits are stated clearly. The packed path is MPS-only for now, architecture coverage is the Qwen3.5 dense and mixture-of-experts models plus compatible Qwen3.8 checkpoints, and padded batches cannot use the mask shortcut and can be slower. Two of the changes landed in generate itself, so every model in the library benefits: an unnecessary attention mask is dropped early, and the stopping check is read one step late so the processor keeps scheduling work.