LLMs, GenAI, RAG, agents, evaluation · insight-0505

Separate total from active parameters when sizing mixture-of-experts models

Situation
A team compares a mixture-of-experts (MoE) model with a dense model for self-hosting using only the headline parameter count.
Action
Record two numbers per candidate, total parameters (weight memory) and parameters active per input (compute per step), and size hardware from both plus the key-value cache.
Reason
Sparse MoE layers route each input to a subset of experts, so parameter count grows without a matching rise in computation per input.
Boundary
Self-hosted or on-premise deployment plans, model comparison tables, system-design interviews.

Symptom or risk

They underestimate GPU memory, because all experts must be loaded, or overestimate per-token compute, so the hardware plan is wrong.

For hosted APIs memory is the provider's concern; latency and quality on your eval set matter more.

MoE offers more capacity per unit of compute but needs more memory and more complex serving.

Using either parameter count as a quality proxy across architectures instead of measuring on your task.

Switch Transformers describes mixture-of-experts models that select different parameters for each input, giving a very large parameter count at a constant computational cost.
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity