Tokenization and text representation · MLD-1335
WordPiece
A subword tokenization scheme used by BERT-family models, commonly decoded using longest matching vocabulary pieces.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Split words into vocabulary subwords.
Original caveat
Training and segmentation details differ from BPE.
Context and aliases
Level: Intermediate. Stages: Data · Training · Inference.
Tags: NLP · SLM · LLM.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Hugging Face: chat templates ↗ · Reference link
- Transformers glossary ↗ · Consulted
- Hugging Face: tokenization algorithms ↗ · Consulted
- Original reference URL ↗