Sequence architectures and attention · MLD-0498
Grouped-query attention
Attention where groups of query heads share key and value heads.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Balance attention quality and decoding memory.
Original caveat
The number of KV heads is distinct from query-head count.
Context and aliases
Level: Advanced. Stages: Training · Inference · Theory.
Tags: DL · SLM · LLM · NLP.
Aliases: GQA.
Related concepts
Concepts linking here
Original references
- GQA: Training Generalized Multi-Query Transformer Models ↗ · Reference link
- Transformers glossary ↗ · Consulted
- Original reference URL ↗