Tags
No item matches the filter.
Build b11556 adds the weight rotation step that Bonsai 2 27B was missing in stock llama.cpp. The 7.63 GB Q2_0 file now gives correct output there, while PrismML's smaller PQ2_0 and PTQ1_0 packs still require the company's own build.
llama.cppverified
A GGUF checkpoint can be passed straight to from_pretrained and run with its weights still packed, using ggml's Metal kernels through the kernels library. The target is a single interactive conversation on an Apple Silicon Mac, and the packed path covers the Qwen3.5 dense and mixture-of-experts architectures plus compatible Qwen3.8 checkpoints. Hugging Face says the result comes close to llama.cpp on the three checkpoints it measured, while noting the two measurements were not taken under identical conditions. It still recommends llama.cpp when efficient local inference is the priority.
Hugging Faceverified
The 0.4.0 release landed on September 4, 2026, and it is a numbered version rather than one of the nightly builds. It adds initial support for Qwen3.8-Flash-Next, NVIDIA Nemotron-3-Puzzle-75B-A9B, DSpark on Nemotron 3.5, and nanbeige4.2-3B. Apple RDMA becomes an RPC transport, video arrives as command-line options, and the quantizer gains a memory ceiling. Two server defaults changed, and saved session state carries a new version number.
llama.cppverified
A local model has one hard requirement before speed, quality or license enter the discussion: the weights have to fit in memory. That turns the machine question into arithmetic you can do on paper, and Apple's August 25 desktops give you three price points to do it against — $899 for a 32GB ceiling, $2,499 for 128GB, $5,499 for 512GB. This guide works through what fits where, which runner to use, what each size class is good for, and how many tokens it takes before a machine beats paying by the token.
Apple, OpenRouter, DeepSeekverified
Qwen3.8-Flash-Next is an open-weights multimodal mixture-of-experts model and an early preview of the Qwen4 architecture. The interesting pair of numbers is 125B parameters against 6B activated per token: the first decides whether it fits on your machine, the second decides how fast it answers. Simon Willison has it running on a DGX Spark from quantized files of 72.5GB and 78.9GB, which is the honest hardware requirement.
Qwenverified
Granite 4.2 comes as 3B, 8B and 30B under Apache 2.0, with GGUF and 4-bit builds published alongside the weights. The interesting part is not the benchmark table — it is that the license and the file formats are settled on day one.
IBMverified
