Tokenization and text representation · MLD-0306
Detokenization
Converting tokens or token IDs back into text or another output representation.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Render generated token IDs as text.
Original caveat
The process may not exactly reverse lossy preprocessing.
Context and aliases
Level: Intermediate. Stages: Data · Training · Inference.
Tags: NLP · SLM · LLM.
Aliases: Token decoding.
Related concepts
Original references
- Hugging Face: chat templates ↗ · Reference link
- Transformers glossary ↗ · Consulted
- Hugging Face: tokenization algorithms ↗ · Consulted
- Original reference URL ↗