Tokenization and text representation · MLD-1236

Token

A unit represented by a tokenizer or model, such as a subword, character, byte, or special marker.

Expanded guide · editorial review 2026-11-01. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.

Prerequisites

How it works

A token is an element in a model's input or output vocabulary. Text tokenizers can use subwords, characters or bytes, with special tokens for structure. Token IDs index the vocabulary; the numeric distance between IDs has no semantic meaning. The tokenizer is part of the model's interface and must match its training conventions.

Worked example

A hypothetical tokenizer splits 'unhelpful' into 'un', 'help' and 'ful'. Another tokenizer might choose different pieces or one whole-word token. The example is illustrative: obtain actual IDs and boundaries from the exact tokenizer rather than assuming a universal split.

A useful computation

For a context budget L, reserve space for input tokens, structural tokens and intended output. Count after applying the chat template, because role markers also consume positions.

What to test

Round-trip representative text through encoding and decoding. Include accented characters, emoji, code, whitespace and long numbers. Inspect truncation behavior and special-token handling.

Common pitfall

Words, characters and tokens are different units. A fixed word-to-token ratio can fail badly across languages and content types.

Sources for the expanded explanation

Editorial synthesis checked against the linked primary material; no independent replication is claimed.

Original example

Split text into model input units.

Original caveat

A token is not necessarily a word.

Context and aliases

Level: Foundational. Stages: Data · Training · Inference.

Tags: NLP · SLM · LLM.

Aliases: None recorded.

Related concepts

Concepts linking here

Original references

Open original record, export or annotate →