AI safety and alignment · MLD-0470
Goodhart's law
The principle that optimizing a measure can weaken its usefulness as a measure of the intended target.
Expanded guide · editorial review 2026-11-01. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Prerequisites
Mechanism: selection amplifies proxy error
Goodhart effects arise when a variable initially useful for predicting the real target becomes less reliable under strong selection or intervention. A measurable proxy P can combine target quality Q with noise or unrelated features. Selecting extreme P values tends to select favorable noise as well as high Q; the effect depends on the proxy, environment and strength of optimization.
A small mathematical illustration
Suppose P = Q + E, where Q is the quality you care about and E is zero-mean measurement error. Picking the highest P from many candidates tends to pick candidates with unusually positive E. A higher maximum P need not imply an equally higher Q. This toy model illustrates regressional Goodhart; it is not a universal formula for every failure.
Four mechanisms, not one theorem
Manheim and Garrabrant distinguish regressional effects (selecting noise), extremal effects (leaving the region where correlations were learned), causal effects (intervening on the metric does not cause the target to improve) and adversarial effects (other agents adapt to the target metric). Real systems can combine multiple mechanisms.
Worked AI-safety example
Imagine a code-generation benchmark in which a learned reviewer assigns scores based on formatting and a handful of tests. Increasing best-of-N selection may raise reviewer scores while independently verified functional correctness plateaus or falls. This is an illustrative experiment to run, not a measured claim about Atlas, your models or any particular judge.
Test design: diagnose the break point
For increasing selection budgets N = 1, 4, 16, 64, freeze the candidate generator and judge. Record the chosen candidate's proxy score, separate hidden-test score and uncertainty over repeated seeds. Plot target score against proxy score and inspect error clusters. Audit whether evaluator leakage or changed data distributions confound the result.
Tradeoffs and misconceptions
Goodhart's law does not imply metrics are worthless or that optimization always reduces true quality. Often both metrics improve at first; failure may emerge only under stronger pressure or distribution change. Improve metric robustness, maintain independent outcomes and avoid declaring a causal mechanism from correlation alone.
Sources for the expanded explanation
- Manheim and Garrabrant, Categorizing Variants of Goodhart's Law (2018) ↗
- OpenAI, Scaling laws for reward model overoptimization (2022) ↗
Editorial synthesis checked against the linked primary material; no independent replication is claimed.
Original example
Selection for judge scores stops improving real correctness.
Original caveat
It describes a family of failure mechanisms, not one precise universal theorem.
Context and aliases
Level: Advanced. Stages: Training · Evaluation · Research · Governance.
Tags: Safety · RL · SLM · LLM.
Aliases: None recorded.