Optimization and training dynamics · MLD-0768
Momentum
An accumulated direction that smooths and accelerates gradient-based updates.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Reduce oscillation in SGD.
Original caveat
Momentum is not merely a larger learning rate.
Context and aliases
Level: Intermediate. Stages: Training · Theory.
Tags: ML · DL · SLM · LLM.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Deep Learning: online textbook ↗ · Consulted
- PyTorch optimization algorithms ↗ · Reference link
- PyTorch documentation ↗ · Reference link
- Original reference URL ↗