Tokenization and text representation · MLD-0684
Loss masking
Excluding selected positions from the training loss.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Train only on assistant response tokens.
Original caveat
Loss masking does not prevent those positions from influencing later activations.
Context and aliases
Level: Intermediate. Stages: Data · Training · Inference.
Tags: NLP · SLM · LLM.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Hugging Face: chat templates ↗ · Reference link
- Transformers glossary ↗ · Consulted
- Hugging Face: tokenization algorithms ↗ · Consulted
- Original reference URL ↗