Sequence architectures and attention · MLD-1239
Token dropping
Skipping or rerouting tokens that exceed an expert's capacity in some MoE systems.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Handle overloaded experts during training.
Original caveat
Dropping can affect quality and is not used by all MoE designs.
Context and aliases
Level: Intermediate. Stages: Training · Inference · Theory.
Tags: DL · SLM · LLM · NLP.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Attention Is All You Need ↗ · Consulted
- Transformers glossary ↗ · Consulted
- PyTorch neural-network layers ↗ · Reference link
- Original reference URL ↗