Optimization and training dynamics · MLD-0023
Adafactor
An adaptive optimizer that can factor second-moment estimates to reduce optimizer-state memory.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Train large models with less optimizer memory.
Original caveat
Factored estimates are an approximation with shape-dependent behavior.
Context and aliases
Level: Intermediate. Stages: Training · Theory.
Tags: ML · DL · SLM · LLM.
Aliases: None recorded.
Related concepts
Original references
- Deep Learning: online textbook ↗ · Consulted
- PyTorch optimization algorithms ↗ · Reference link
- PyTorch documentation ↗ · Reference link
- Original reference URL ↗