Systems and efficient execution · MLD-0605
Key-value cache
Stored attention keys and values from earlier tokens that avoid recomputing those states during generation.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Accelerate autoregressive transformer inference.
Original caveat
Cache memory grows with context, batch size, layers, and representation size.
Context and aliases
Level: Advanced. Stages: Training · Inference · Deployment.
Tags: DL · LLM · SLM · Systems.
Aliases: KV cache.
Related concepts
Concepts linking here
Technology connections
- BentoML — Model packaging and inference-service framework.
- KServe — Kubernetes-native model inference platform.
- NVIDIA Triton Inference Server — Multi-framework inference serving platform.
- ONNX Runtime — Cross-platform execution engine for ONNX and supported model formats.
- Ray Serve — Scalable model-serving library on Ray.
- SGLang — Serving runtime for language and multimodal models.
- TensorRT — NVIDIA optimizer and inference runtime for neural networks.
- TensorRT-LLM — NVIDIA optimization and execution stack for LLM inference.
- Text Generation Inference (TGI) — Hugging Face text-generation server now archived/maintenance-oriented.
- vLLM — Inference engine and API server for supported language and multimodal models.
Original references
- PyTorch Fully Sharded Data Parallel ↗ · Consulted
- PyTorch documentation ↗ · Reference link
- vLLM documentation ↗ · Consulted
- Original reference URL ↗