Sequence architectures and attention · MLD-1264
Transformer
A neural architecture built around attention and feed-forward blocks rather than requiring recurrent sequence processing.
Expanded guide · editorial review 2026-11-01. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Prerequisites
How it works
A Transformer combines attention, position information and feedforward transformations. Attention mixes information across sequence positions; the feedforward block transforms each position's representation. Residual connections and normalization help information and gradients pass through stacked blocks. Encoder, decoder and encoder-decoder variants use different visibility patterns.
Worked example
For translation, an encoder represents the source sentence and a decoder generates the target sentence. The decoder can attend to encoded source positions while a causal mask prevents it from reading future target tokens. A decoder-only language model uses a different arrangement, so the original translation architecture should not be assumed for every LLM.
A useful computation
One conceptual block is attention mixing followed by a position-wise feedforward network, with residual paths around the sublayers. Exact normalization order varies across implementations.
What to test
Check mask correctness with tiny sequences. Measure memory and time as sequence length increases; dense pairwise attention creates a quadratic score matrix.
Common pitfall
Positional information and causal masking are separate mechanisms. Adding positions does not prevent information leakage from future tokens.
Sources for the expanded explanation
Editorial synthesis checked against the linked primary material; no independent replication is claimed.
Original example
Model text, image patches, or other token sequences.
Original caveat
Not all language models are Transformers.
Context and aliases
Level: Intermediate. Stages: Training · Inference · Theory.
Tags: DL · SLM · LLM · NLP.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Attention Is All You Need ↗ · Consulted
- Transformers glossary ↗ · Consulted
- Original reference URL ↗