Sequence architectures and attention · MLD-0069
Attention head
One attention computation with its own learned projections or subspace.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Learn different contextual relationships in parallel.
Original caveat
Heads can be redundant or distributed in function.
Context and aliases
Level: Intermediate. Stages: Training · Inference · Theory.
Tags: DL · SLM · LLM · NLP.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Attention Is All You Need ↗ · Consulted
- Transformers glossary ↗ · Consulted
- PyTorch neural-network layers ↗ · Reference link
- Original reference URL ↗