Tags
No item matches the filter.
Build b11556 adds the weight rotation step that Bonsai 2 27B was missing in stock llama.cpp. The 7.63 GB Q2_0 file now gives correct output there, while PrismML's smaller PQ2_0 and PTQ1_0 packs still require the company's own build.
llama.cppverified
Google's AI Edge team released ML Drift under the Apache 2.0 license on October 8, 2026. It is the engine that runs on-device models on the GPU in LiteRT, and it also ships as a standalone library. The older TFLite GPU delegate gets no new features from now on, so apps built on it face a migration.
Google Developers Blogverified
Google DeepMind released EmbeddingGemma 2 on October 6, 2026, under Apache 2.0. The 740 million parameter model maps text, code, images, video, and audio into one embedding space and is built to run on a phone. It keeps the multilingual text quality of the first version, raises the MTEB Code score by almost ten points, and is already supported by llama.cpp, Ollama, MLX, vLLM, and transformers.js.
Google DeepMindverified
The llama.cpp project released version 0.6.0 on October 5, 2026. It adds support for GLM-5.3-Flash, a model for text and images with 320 billion parameters, and a new server endpoint for decision models. Two changes to speculative decoding come with measurements from their authors, one on Nvidia's DGX Spark and one on Apple GPUs. The release also changes the C API and raises the version of the saved session format, so programs built on the library need a check before you upgrade.
llama.cppverified
Strands Labs, the experimental arm of the Strands Agents project, released Strands Decider 2B on October 1, 2026. The model does not write text: it picks one of the options you give it, answers yes or no, or scores on a scale, and it attaches a confidence value to each answer. The team reports a median of 115 ms per decision on Nvidia's RTX 3090. Code, weights, training data, and scripts are public under Apache-2.0. If you build agents, this gives you a cheap local step for routing, tool checks, and guardrails.
Strands Agentsverified
Google added local model support to the Antigravity SDK on September 23, 2026. The first supported setup is Gemma 4 26B A4B running through LiteRT, and Google recommends more than 24 GB of video or unified memory. The SDK can also talk to local servers such as Ollama, LM Studio, and vLLM.
Google Developers Blogverified
The llama.cpp project released version 0.5.0 on September 23, 2026. It adds support for HRM-Text, the architecture behind DFM Mimir 1B, and conversion for MiMo-V2.6. The server can now listen on several addresses at once, and router mode no longer passes the API key file to the model processes it starts.
llama.cppverified
A GGUF checkpoint can be passed straight to from_pretrained and run with its weights still packed, using ggml's Metal kernels through the kernels library. The target is a single interactive conversation on an Apple Silicon Mac, and the packed path covers the Qwen3.5 dense and mixture-of-experts architectures plus compatible Qwen3.8 checkpoints. Hugging Face says the result comes close to llama.cpp on the three checkpoints it measured, while noting the two measurements were not taken under identical conditions. It still recommends llama.cpp when efficient local inference is the priority.
Hugging Faceverified
Build b11077, published on September 21, 2026, fixes an authentication bug in llama.cpp's router mode. When a server was started with both a key on the command line and a file of keys, the child instances it spawned validated against the file only, so a client using the command-line key could list models but got 401 on every chat request. The same change stops the key file path from reaching a child process at all. It is one line in one file, and it is the difference between a working deployment and one that looks healthy while refusing work.
llama.cppverified
Google opened pre-orders for Googlebook on September 21, 2026, with prices starting at $899 and shipping from October 4, 2026. The laptops run an Android-based system with desktop foundations from ChromeOS, on Intel and Qualcomm chips paired with NPUs rated at over 45 TOPS. For anyone who writes code, the notable part is not the assistant features but a pKVM hypervisor with a Level 5 security certification, which powers an isolated Linux environment with full terminal access. Neither announcement says which Gemini features run on the device and which run in the cloud.
Googleverified
Hugging Face published the release candidate of tokenizers v1 on September 21, 2026, together with the numbers behind it. Across the ten model families the new encode path covers, the library encodes 3 to 30 times faster than v0.23 on a single thread of an Apple M4 Max, and it reaches 76% of linear scaling across eight workers. The output does not change: the same text produces the same token IDs, through the same calls. Every figure was measured against the Rust crate, and the Python bindings most of the ecosystem uses add per-call overhead the numbers exclude.
Hugging Faceverified
Mac Studio with M5 Max and the all-new M5 Ultra reaches stores on September 22, 2026, and Apple puts its AI performance at up to 4.3x the previous generation, measured in July 2026 against a machine with M3 Ultra. The configuration that decides how large a local model can be, 512GB of unified memory at up to 1.2TB/s, sits in a footnote: it comes in late October. On the software side the new Siri AI ships in beta, in English to start, and will not be available in the EU on iOS, iPadOS or watchOS at launch. Apple Intelligence features that run on server-side models also have daily usage limits, and Apple says wider access to them will be sold for a fee later.
Appleverified
NVIDIA published the Personal AI Router at IFA 2026 on September 3, 2026. It is a free, open source tool for Windows, macOS, and Linux that finds compatible machines on your local network and routes each independent inference request to one that has capacity, so parallel agent subtasks stop queueing behind a single GPU. It works with Ollama and LM Studio, and supports GeForce RTX 20 Series and newer, RTX PRO workstation cards, DGX Spark, and Apple M4 silicon. What it does not do is pool memory, so a model that does not fit on one machine still does not fit.
NVIDIAverified
The @huggingface/kernels package downloads one GPU operation from the Hub and runs it in a browser that supports WebGPU. All 207 kernels are Apache-2.0, and each one is published with its interface contract, correctness cases, and benchmark cases in the same repository. Hugging Face compared them with ONNX Runtime Web 1.30.0 on an Apple M4 GPU and reports a 2.57x geometric mean speedup across 809 comparable cases. The timings cover work on the GPU alone, for single operations rather than whole models.
Hugging Faceverified
Between August 29 and 31, 2026, llama.cpp shipped a run of changes that all point at the same place: the part of inference that gets slower as the context grows. Vulkan, CUDA and the Metal and OpenCL paths each got their own fix, with numbers the authors published. Three crashes were closed alongside them. None of it needs a flag; it arrives with the next build.
llama.cppverified
A local model has one hard requirement before speed, quality or license enter the discussion: the weights have to fit in memory. That turns the machine question into arithmetic you can do on paper, and Apple's August 25 desktops give you three price points to do it against — $899 for a 32GB ceiling, $2,499 for 128GB, $5,499 for 512GB. This guide works through what fits where, which runner to use, what each size class is good for, and how many tokens it takes before a machine beats paying by the token.
Apple, OpenRouter, DeepSeekverified
Qwen3.8-Flash-Next is an open-weights multimodal mixture-of-experts model and an early preview of the Qwen4 architecture. The interesting pair of numbers is 125B parameters against 6B activated per token: the first decides whether it fits on your machine, the second decides how fast it answers. Simon Willison has it running on a DGX Spark from quantized files of 72.5GB and 78.9GB, which is the honest hardware requirement.
Qwenverified
Granite 4.2 comes as 3B, 8B and 30B under Apache 2.0, with GGUF and 4-bit builds published alongside the weights. The interesting part is not the benchmark table — it is that the license and the file formats are settled on day one.
IBMverified
The new Mac Studio goes to 512GB of unified memory at $5,499, though that configuration does not arrive until late October; the new Mac mini stops at 32GB and starts at $899. For anyone running models on their own machine, that gap is the whole product line, and Apple is now selling it that way.
Appleverified
