Decoding and inference behavior · MLD-1280
Typical sampling
Selecting tokens according to how close their information content is to the distribution's expected information.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Encourage locally typical generated continuations.
Original caveat
Typicality differs from simply taking the most probable tokens.
Context and aliases
Level: Intermediate. Stages: Inference · Evaluation · Deployment.
Tags: SLM · LLM · NLP.
Aliases: None recorded.
Related concepts
Original references
- Hugging Face: generation strategies ↗ · Reference link
- Fast Inference from Transformers via Speculative Decoding ↗ · Consulted
- vLLM documentation ↗ · Consulted
- Original reference URL ↗