Systems and efficient execution · MLD-0267
Data parallelism
Running model replicas on different data partitions and combining their learning signals.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Train on multiple GPUs with separate mini-batches.
Original caveat
Global batch size and synchronization influence optimization.
Context and aliases
Level: Advanced. Stages: Training · Inference · Deployment.
Tags: DL · LLM · SLM · Systems.
Aliases: DP training.
Related concepts
Concepts linking here
Technology connections
- DeepSpeed — Training and inference optimization framework with ZeRO.
- Hugging Face Accelerate — Abstraction for launching and adapting training across device configurations.
- Lightning Fabric — Lightweight training infrastructure around PyTorch loops.
- Megatron Core — Building blocks for large Transformer training.
- PyTorch DistributedDataParallel — Synchronized replicated-model training across workers.
- PyTorch FSDP2 — Composable fully sharded data parallel training.
- Ray Train — Distributed training orchestration within Ray.
- verl — Flexible reinforcement-learning framework for language models.
Original references
- PyTorch Fully Sharded Data Parallel ↗ · Consulted
- PyTorch documentation ↗ · Reference link
- vLLM documentation ↗ · Consulted
- Original reference URL ↗