Regularization and generalization · MLD-1325
Weight decay
An update mechanism shrinking weights toward zero.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Apply decoupled decay in AdamW.
Original caveat
Bias and normalization parameters are often handled separately.
Context and aliases
Level: Intermediate. Stages: Theory · Training · Evaluation.
Tags: ML · DL · SLM · LLM.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Deep Learning: online textbook ↗ · Consulted
- Google Machine Learning Glossary ↗ · Consulted
- PyTorch neural-network layers ↗ · Reference link
- Original reference URL ↗