Sequence architectures and attention · MLD-0049
ALiBi
Adding head-specific linear position-distance biases to attention scores.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Encourage useful behavior beyond training lengths.
Original caveat
It does not guarantee arbitrary context-length generalization.
Context and aliases
Level: Intermediate. Stages: Training · Inference · Theory.
Tags: DL · SLM · LLM · NLP.
Aliases: Attention with linear biases.
Related concepts
Concepts linking here
Original references
- Train Short, Test Long: Attention with Linear Biases ↗ · Reference link
- Transformers glossary ↗ · Consulted
- Original reference URL ↗