Deployment and MLOps · MLD-0764
Model serving
Making a model available for prediction through a process, service, or application interface.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Expose a prediction endpoint.
Original caveat
A functioning endpoint is not proof of quality or reliability.
Context and aliases
Level: Intermediate. Stages: Deployment · Monitoring.
Tags: ML · DL · LLM · SLM · MLOps.
Aliases: None recorded.
Related concepts
Concepts linking here
Technology connections
- BentoML — Model packaging and inference-service framework.
- KServe — Kubernetes-native model inference platform.
- NVIDIA Triton Inference Server — Multi-framework inference serving platform.
- ONNX Runtime — Cross-platform execution engine for ONNX and supported model formats.
- Ray Serve — Scalable model-serving library on Ray.
- SGLang — Serving runtime for language and multimodal models.
- TensorRT — NVIDIA optimizer and inference runtime for neural networks.
- TensorRT-LLM — NVIDIA optimization and execution stack for LLM inference.
- Text Generation Inference (TGI) — Hugging Face text-generation server now archived/maintenance-oriented.
- vLLM — Inference engine and API server for supported language and multimodal models.
Original references
- Transformers glossary ↗ · Consulted
- Google Cloud MLOps architecture ↗ · Reference link
- scikit-learn User Guide ↗ · Consulted
- Original reference URL ↗