Tokenization and text representation · MLD-1242
Tokenization
Converting raw text or other input into discrete units and their identifiers.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Prepare a prompt for a language model.
Original caveat
Different tokenizers produce different token counts.
Context and aliases
Level: Foundational. Stages: Data · Training · Inference.
Tags: NLP · SLM · LLM.
Aliases: None recorded.
Related concepts
Concepts linking here
Technology connections
- Gensim — Topic modeling and vector-space text-analysis library.
- Hugging Face Datasets — Dataset loading, transformation, caching and streaming toolkit.
- Hugging Face Diffusers — Reusable pipelines and components for diffusion-based generative models.
- Hugging Face Tokenizers — Fast tokenizer training and execution library with Rust internals.
- Hugging Face Transformers — Common interfaces and implementations for pretrained model architectures.
- OpenCLIP — Open implementation and training tooling for CLIP-style contrastive models.
- Sentence Transformers — Framework for embedding and reranking text and other supported inputs.
- SentencePiece — Language-independent subword tokenizer and trainer.
- spaCy — Production-oriented NLP pipelines and linguistic annotations.
- timm — PyTorch image-model collection with training utilities and pretrained weights.
Original references
- Hugging Face: chat templates ↗ · Reference link
- Transformers glossary ↗ · Consulted
- Hugging Face: tokenization algorithms ↗ · Consulted
- Original reference URL ↗