Systems and efficient execution · MLD-0844
Online inference
Producing predictions in response to arriving requests within an operational latency budget.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Answer an interactive application request.
Original caveat
Online inference does not imply online model training.
Context and aliases
Level: Advanced. Stages: Training · Inference · Deployment.
Tags: DL · LLM · SLM · Systems.
Aliases: Real-time inference.
Related concepts
Concepts linking here
Original references
- PyTorch Fully Sharded Data Parallel ↗ · Consulted
- PyTorch documentation ↗ · Reference link
- vLLM documentation ↗ · Consulted
- Original reference URL ↗