Tokenization and text representation · MLD-1228
Text normalization
Applying chosen canonicalization rules to text before processing.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Normalize Unicode forms or whitespace.
Original caveat
Normalization may remove distinctions important to a task.
Context and aliases
Level: Intermediate. Stages: Data · Training · Inference.
Tags: NLP · SLM · LLM.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Hugging Face: chat templates ↗ · Reference link
- Transformers glossary ↗ · Consulted
- Hugging Face: tokenization algorithms ↗ · Consulted
- Original reference URL ↗