llama.cpp spent two days on long context, and the gains landed on three different backends

Read one at a time, the past three days of llama.cpp look like a changelog. Read together, they are one piece of work: long context is where local inference gets expensive, and four separate changes went after it on four different backends.

Long context, three fixes

The largest single number comes from one helper function. It scanned cache cells whose count grew with the context, and that work is now gone. On an RTX PRO 6000 running Qwen3.8-Flash-Next at 132,000 tokens of context, generation went from 33.6 to 50.9 tokens per second, shipped in b10707. Prompt processing is unchanged, and on short prompts nothing shows at all.

Vulkan got a radix sort for top-k selection, covering k of 1024 and above, which is what Qwen 3.8 Flash Next asks for. On a DGX Spark at 32,768 tokens of context, prompt processing went from 390.1 to 520.8 tokens per second — past the 498.5 measured on CUDA on the same machine. At zero context depth the same code is slightly slower, so the gain belongs to long context and nothing else.

On CUDA, the helper that assigns warp threads to tokens took a shortcut only when the number of active experts divided the warp evenly, with a single hand-written exception padding 6 up to 8. Generalizing that padding to the next power of two hands the shortcut to 10 as well. On the same RTX PRO 6000 at 55,000 tokens, prompt processing rose from 2,334 to 2,600 tokens per second in b10704.

Small batches on AMD

Two constants in Vulkan's matrix-vector path were retuned for RDNA3 cards, and the gain starts at a batch of five columns. In the author's micro-benchmark the four-bit quantized types gain the most, up to 46.2% at eight columns. The change is conditioned on AMD hardware of that architecture, and the author had a single Strix Halo machine, so whether the same numbers appear on other cards of the generation is open.

Phone silicon got its own pass

On Adreno X2, OpenCL now defaults to the faster matmul path. The author of the change measures prompt processing on gpt-oss-20b at about 25% faster, with a second change worth roughly 9% on gemma-3n-E4B. Token generation is unchanged in both cases. Both shipped in b10687.

The Hexagon backend stopped reading Qualcomm's NPU core count from a hard-coded table and now asks the chip at runtime, in b10693. The same change stops reserving NPU sessions while the backend starts, and reserves them only once a model is actually loaded. Of the devices the author tested, exactly one phone answers the new query, so everything else still falls back to the written-in values.

Three crashes closed

A bug report on August 29 showed llama.cpp crashing on Apple's Metal backend when a quantized weight met an F16 operand on the single-token path. The kernel that shape needs was never written, and the code used the null pointer it got back instead of refusing the work. Fixed in b10691.

A second crash, on context shift with an unquantized K cache, closed in b10690 the same morning. Separately, a client could not drive several RPC servers at once: pointers to remote buffers were sent even when the buffer belonged to a different server, which then received an address that means nothing to it and refused to build the graph. The fix sends the pointer only when the buffer belongs to that connection, and adds a two-server test, because the bug does not exist with one.