Sequence architectures and attention · MLD-0742
Mixture of experts
A model combining specialized subnetworks through a learned or specified routing mechanism.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Activate selected expert feed-forward modules per token.
Original caveat
MoE can be dense or sparse; it is not necessarily several independent chatbots.
Context and aliases
Level: Advanced. Stages: Training · Inference · Theory.
Tags: DL · SLM · LLM · NLP.
Aliases: MoE.