Optimization and training dynamics · MLD-0850
Optimizer state
Persistent values used by an optimizer, such as momentum and second-moment estimates.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Store Adam's moving averages.
Original caveat
Optimizer state can consume more memory than model weights.
Context and aliases
Level: Intermediate. Stages: Training · Theory · Inference · Deployment.
Tags: ML · DL · SLM · LLM · Systems.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Deep Learning: online textbook ↗ · Consulted
- PyTorch Fully Sharded Data Parallel ↗ · Consulted
- PyTorch optimization algorithms ↗ · Reference link
- PyTorch documentation ↗ · Reference link
- vLLM documentation ↗ · Consulted
- Original reference URL ↗