Mixture of experts (MoE)
An architecture that holds many parameters but uses only a small share of them for each token, so a large model can answer at the cost of a much smaller one.
The layers are split into groups, and a router picks a couple of them per token. Total size and working size therefore stop being the same number, which is why announcements now quote both — the first says what the model knows, the second what each answer costs to compute.
It changes what a parameter count tells you. Comparing a dense model with a mixture by total parameters alone compares the wrong quantity; the active share is what sets speed and price.
For running one yourself the catch is memory. All the parameters have to be resident even though only a fraction is used at a time, so a mixture is cheap to run and expensive to hold.