Mathematics and probability · MLD-0369
Entropy
Expected uncertainty or information content under a probability distribution.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Measure how spread out token probabilities are.
Original caveat
High entropy does not necessarily mean useful diversity.
Context and aliases
Level: Foundational. Stages: Theory · Training · Evaluation.
Tags: ML · DL · SLM · LLM.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Deep Learning: online textbook ↗ · Consulted
- Mathematics for Machine Learning ↗ · Reference link
- Original reference URL ↗