Foundations · MLD-0863

Overfitting

Learning training-specific detail that fails to transfer to new examples.

Expanded guide · editorial review 2026-11-01. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.

Prerequisites

How it works

Overfitting occurs when choices adapt too closely to observed training or validation data and perform worse on genuinely unseen cases. Capacity, noise, small samples and repeated model selection can contribute. A training-validation gap is a warning signal; distribution shift and leakage can produce misleading patterns too.

Worked example

Imagine a classifier achieving 99% on training tickets and 73% on later tickets. A simpler model scores 88% and 82%. Those invented numbers illustrate why training accuracy alone cannot choose a model. Investigate duplicate tickets, temporal drift and label quality before attributing the entire gap to model capacity.

A useful procedure

Fit on training data, tune on validation data and evaluate the chosen configuration once on a reserved test set. Keep correlated users or documents in the same split.

What to test

Use learning curves and suitable cross-validation. Compare regularization, early stopping and additional representative data under the same evaluation protocol.

Common pitfall

Repeatedly choosing the best configuration on the test set makes that set part of model selection and weakens the independence of the reported score.

Sources for the expanded explanation

Editorial synthesis checked against the linked primary material; no independent replication is claimed.

Original example

Memorize noise in a tiny training set.

Original caveat

Low training error alone does not diagnose it.

Context and aliases

Level: Foundational. Stages: Theory · Training · Inference.

Tags: ML · DL · SLM · LLM.

Aliases: None recorded.

Related concepts

Concepts linking here

Original references

Open original record, export or annotate →