Skip to content

Glossary

Quantization

Storing a model's weights at lower precision so it fits in less memory — the usual way a model that needs a data-centre card comes to run on a laptop.

Weights normally sit at sixteen bits each. Cut them to four and the same model needs roughly a quarter of the memory, which is the difference between a seven-billion-parameter model asking for around fourteen gigabytes and asking for around four.

The loss is real but uneven. Ordinary conversation survives four-bit almost intact; arithmetic, long chains of reasoning, and code degrade first and quietly — the model does not announce that it has become worse, it just starts being wrong more often.

This is what makes open-weights models practically usable outside a data centre, and it is why a local model should be measured on your own task before it is trusted, not on someone else's benchmark run at full precision.