Sequence architectures and attention · MLD-0649
Linear attention
Attention-like mechanisms rearranged or approximated to avoid quadratic sequence-length cost.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Process long sequences with kernelized aggregation.
Original caveat
It is not generally identical to exact softmax attention.
Context and aliases
Level: Intermediate. Stages: Training · Inference · Theory.
Tags: DL · SLM · LLM · NLP.
Aliases: None recorded.
Related concepts
Original references
- Attention Is All You Need ↗ · Consulted
- Transformers glossary ↗ · Consulted
- PyTorch neural-network layers ↗ · Reference link
- Original reference URL ↗