Systems and efficient execution · MLD-1342
ZeRO
A family of distributed training optimizations that partitions optimizer state, gradients, and optionally parameters across workers.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Reduce duplicated training-state memory.
Original caveat
Different stages shard different state categories.
Context and aliases
Level: Advanced. Stages: Training · Inference · Deployment.
Tags: DL · LLM · SLM · Systems.
Aliases: Zero Redundancy Optimizer.