Systems and efficient execution · MLD-0685
Loss scaling
Multiplying the loss before backpropagation and compensating in gradients to avoid numerical underflow in low-precision training.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Stabilize float16 gradient computation.
Original caveat
Scaling must be reversed before applying the optimizer update.
Context and aliases
Level: Advanced. Stages: Training · Inference · Deployment.
Tags: DL · LLM · SLM · Systems.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- PyTorch Fully Sharded Data Parallel ↗ · Consulted
- PyTorch documentation ↗ · Reference link
- vLLM documentation ↗ · Consulted
- Original reference URL ↗