Tokenization and text representation · MLD-1108
SentencePiece
A tokenization toolkit supporting models such as BPE and Unigram directly over raw text.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Train a tokenizer without presegmented word inputs.
Original caveat
SentencePiece is a library/framework, not one unique algorithm.
Context and aliases
Level: Intermediate. Stages: Data · Training · Inference.
Tags: NLP · SLM · LLM.
Aliases: None recorded.
Related concepts
Concepts linking here
Technology connections
- SentencePiece — Language-independent subword tokenizer and trainer.
Original references
- Hugging Face: chat templates ↗ · Reference link
- Transformers glossary ↗ · Consulted
- Hugging Face: tokenization algorithms ↗ · Consulted
- Original reference URL ↗