Systems and efficient execution · MLD-1003
Reduce-scatter
A collective operation that reduces contributions across workers and distributes different result portions to them.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Store only each worker's reduced gradient shard.
Original caveat
It combines reduction with partitioned output ownership.
Context and aliases
Level: Advanced. Stages: Training · Inference · Deployment.
Tags: DL · LLM · SLM · Systems.
Aliases: None recorded.
Related concepts
Original references
- PyTorch Fully Sharded Data Parallel ↗ · Consulted
- PyTorch documentation ↗ · Reference link
- vLLM documentation ↗ · Consulted
- Original reference URL ↗