Mixture of experts explained
How routers select expert networks, how their outputs combine, and why active parameter counts tell only part of the story about AI memory and speed.
Mixture of experts, or MoE, is a way of organising an AI model so that each input uses a selected part of its internal networks. In a sparse MoE language model, a learned router chooses a few expert networks to process each token. Their outputs are combined, and processing continues through the model.
A token is a piece of text, which may be a word, part of a word or punctuation. Inside the model it has a numerical representation. An expert transforms those numbers. A parameter is a learned numerical weight used in the calculation. Keeping these three things separate makes model specifications much easier to read.
What the router does
Context is the information available to the model for the current task. In the type of transformer model shown here, attention mixes information from the available context. A feed-forward stage then transforms each token’s numerical state. A sparse MoE layer provides a pool of expert feed-forward networks and routes the token through a subset of them.
- Read the token’s current numerical state. This already reflects its context and the processing done so far.
- Score the experts with a learned router. Select the highest-scoring ones. In top-k routing, k is the number selected.
- Run the selected expert networks on that state. The others in the pool do not perform this expert calculation for that token.
- Scale each selected output by its routing weight, then add the outputs together.
- Continue through the model. A later layer can choose a different route for the updated token state.
The Mixtral paper describes this per-token, per-layer choice. Routing is a decision about the token’s current state, so the same written word need not take one fixed route wherever it appears.
A small worked example of the weighted sum
Imagine four experts, with the router selecting experts 2 and 4. Give their outputs weights of 0.7 and 0.3. To keep the arithmetic visible, suppose each expert returns only two numbers. These values are invented for the explanation, not measurements from a real model.
- Expert 2 returns [2, 8]. Multiplying by 0.7 gives [1.4, 5.6].
- Expert 4 returns [6, 4]. Multiplying by 0.3 gives [1.8, 1.2].
- Add corresponding positions. [1.4, 5.6] + [1.8, 1.2] = [3.2, 6.8].
The combined result is another list of numbers. It continues through the model’s remaining calculations. A real model uses far longer lists and many layers, but the example shows what “weighted sum” means. Each selected expert contributes, and the routing weights set the size of those contributions.
What an expert is
An expert is a neural subnetwork within the model. Its parameters are learned during training. The router also learns how to allocate inputs. The name does not promise a separate maths, legal or coding specialist, and these networks do not each write a complete answer for a committee to judge.
The expert pool belongs to particular parts of the model. Shared components, such as attention, still run. Different MoE architectures can also include shared experts that process every token, alongside experts chosen by a router. Check the architecture rather than assuming every model uses the same arrangement.
- Hugging Face’s guide to MoE models in Transformers
- DeepSeek-V3 technical report, shared and routed experts
Total parameters and active parameters
Mistral’s original Mixtral 8x7B, released in December 2023, provides a useful historical example. It has 46.7 billion parameters in total and uses 12.9 billion per token. Each layer has eight experts, with two selected for a token. The active count includes shared components as well as the selected experts.
The model name does not mean eight independent 7-billion-parameter chatbots. Shared parts explain why multiplying the numbers in its name gives the wrong total. Similarly, selecting two of eight experts does not make the entire model’s active count exactly one quarter of its total.
Total parameters describe the collection of learned values. Active parameters describe the portion used for a token. Neither number is a quantity of memory in bytes or a direct measurement of response speed.
Why less calculation still needs memory
In a conventional dense feed-forward layer, each token uses the same network. Sparse routing can reduce calculation compared with using every expert in a similarly sized pool. The unused experts still have learned weights that the serving system must store and make available when selected.
Memory also depends on how those weights are represented and on space for the running computation. Some systems distribute experts across devices or move weights between memory tiers. Those choices change what fits on one device and add their own transfer costs.
The router must also distribute work well. If many tokens select the same experts, some devices can become busy while others wait. When experts sit on different devices, token states travel to them and their results must come back. Fast arithmetic alone does not remove that communication.
What the architecture tells you about performance
Active parameter count helps explain the work per token. Actual response time also depends on hardware, memory bandwidth, communication, context length and how many requests run together. Throughput measures how much work the system completes over time; latency measures how long a particular request takes. A deployment can improve one without improving the other by the same amount.
We would compare candidate models on the intended task, using representative input lengths and concurrent demand. Measure answer quality, response time and the cost of a successful result. A small active count by itself cannot establish that a model is the cheapest or the fastest choice.
How this relates to RAG and AI costs
MoE describes calculations inside a model. Retrieval-augmented generation, or RAG, searches external information and supplies relevant material for the model to use. The two can work together. Retrieved passages become part of the input, which a MoE model then processes through its own internal routing.
For the surrounding concepts, our RAG guide explains document retrieval, and Tokenomics explains the units of text and the factors behind AI pricing. Model selection is also one part of the workflow assessment covered by our consulting and training service.