hardware, local runners, what fits in memory
What it takes to run a model on hardware you own. Machines and the memory ceiling that decides which model fits, the runners that load it, the quantization that shrinks it, and the setup that makes it usable — with the price of each step, because that is the part comparisons leave out.
No item matches the filter.
Build b11556 adds the weight rotation step that Bonsai 2 27B was missing in stock llama.cpp. The 7.63 GB Q2_0 file now gives correct output there, while PrismML's smaller PQ2_0 and PTQ1_0 packs still require the company's own build.
llama.cppverified
Google's AI Edge team released ML Drift under the Apache 2.0 license on October 8, 2026. It is the engine that runs on-device models on the GPU in LiteRT, and it also ships as a standalone library. The older TFLite GPU delegate gets no new features from now on, so apps built on it face a migration.
Google Developers Blogverified
The llama.cpp project released version 0.6.0 on October 5, 2026. It adds support for GLM-5.3-Flash, a model for text and images with 320 billion parameters, and a new server endpoint for decision models. Two changes to speculative decoding come with measurements from their authors, one on Nvidia's DGX Spark and one on Apple GPUs. The release also changes the C API and raises the version of the saved session format, so programs built on the library need a check before you upgrade.
llama.cppverified
The llama.cpp project released version 0.5.0 on September 23, 2026. It adds support for HRM-Text, the architecture behind DFM Mimir 1B, and conversion for MiMo-V2.6. The server can now listen on several addresses at once, and router mode no longer passes the API key file to the model processes it starts.
llama.cppverified
A GGUF checkpoint can be passed straight to from_pretrained and run with its weights still packed, using ggml's Metal kernels through the kernels library. The target is a single interactive conversation on an Apple Silicon Mac, and the packed path covers the Qwen3.5 dense and mixture-of-experts architectures plus compatible Qwen3.8 checkpoints. Hugging Face says the result comes close to llama.cpp on the three checkpoints it measured, while noting the two measurements were not taken under identical conditions. It still recommends llama.cpp when efficient local inference is the priority.
Hugging Faceverified
Build b11077, published on September 21, 2026, fixes an authentication bug in llama.cpp's router mode. When a server was started with both a key on the command line and a file of keys, the child instances it spawned validated against the file only, so a client using the command-line key could list models but got 401 on every chat request. The same change stops the key file path from reaching a child process at all. It is one line in one file, and it is the difference between a working deployment and one that looks healthy while refusing work.
llama.cppverified
Google opened pre-orders for Googlebook on September 21, 2026, with prices starting at $899 and shipping from October 4, 2026. The laptops run an Android-based system with desktop foundations from ChromeOS, on Intel and Qualcomm chips paired with NPUs rated at over 45 TOPS. For anyone who writes code, the notable part is not the assistant features but a pKVM hypervisor with a Level 5 security certification, which powers an isolated Linux environment with full terminal access. Neither announcement says which Gemini features run on the device and which run in the cloud.
Googleverified
Mac Studio with M5 Max and the all-new M5 Ultra reaches stores on September 22, 2026, and Apple puts its AI performance at up to 4.3x the previous generation, measured in July 2026 against a machine with M3 Ultra. The configuration that decides how large a local model can be, 512GB of unified memory at up to 1.2TB/s, sits in a footnote: it comes in late October. On the software side the new Siri AI ships in beta, in English to start, and will not be available in the EU on iOS, iPadOS or watchOS at launch. Apple Intelligence features that run on server-side models also have daily usage limits, and Apple says wider access to them will be sold for a fee later.
Appleverified
The 0.4.0 release landed on September 4, 2026, and it is a numbered version rather than one of the nightly builds. It adds initial support for Qwen3.8-Flash-Next, NVIDIA Nemotron-3-Puzzle-75B-A9B, DSpark on Nemotron 3.5, and nanbeige4.2-3B. Apple RDMA becomes an RPC transport, video arrives as command-line options, and the quantizer gains a memory ceiling. Two server defaults changed, and saved session state carries a new version number.
llama.cppverified
NVIDIA published the Personal AI Router at IFA 2026 on September 3, 2026. It is a free, open source tool for Windows, macOS, and Linux that finds compatible machines on your local network and routes each independent inference request to one that has capacity, so parallel agent subtasks stop queueing behind a single GPU. It works with Ollama and LM Studio, and supports GeForce RTX 20 Series and newer, RTX PRO workstation cards, DGX Spark, and Apple M4 silicon. What it does not do is pool memory, so a model that does not fit on one machine still does not fit.
NVIDIAverified
The @huggingface/kernels package downloads one GPU operation from the Hub and runs it in a browser that supports WebGPU. All 207 kernels are Apache-2.0, and each one is published with its interface contract, correctness cases, and benchmark cases in the same repository. Hugging Face compared them with ONNX Runtime Web 1.30.0 on an Apple M4 GPU and reports a 2.57x geometric mean speedup across 809 comparable cases. The timings cover work on the GPU alone, for single operations rather than whole models.
Hugging Faceverified
Between August 29 and 31, 2026, llama.cpp shipped a run of changes that all point at the same place: the part of inference that gets slower as the context grows. Vulkan, CUDA and the Metal and OpenCL paths each got their own fix, with numbers the authors published. Three crashes were closed alongside them. None of it needs a flag; it arrives with the next build.
llama.cppverified
A local model has one hard requirement before speed, quality or license enter the discussion: the weights have to fit in memory. That turns the machine question into arithmetic you can do on paper, and Apple's August 25 desktops give you three price points to do it against — $899 for a 32GB ceiling, $2,499 for 128GB, $5,499 for 512GB. This guide works through what fits where, which runner to use, what each size class is good for, and how many tokens it takes before a machine beats paying by the token.
Apple, OpenRouter, DeepSeekverified
The new Mac Studio goes to 512GB of unified memory at $5,499, though that configuration does not arrive until late October; the new Mac mini stops at 32GB and starts at $899. For anyone running models on their own machine, that gap is the whole product line, and Apple is now selling it that way.
Appleverified
