AI safety and alignment · MLD-0146

Capability failure

Failure because a system cannot successfully perform a task, rather than because it pursues a different objective.

Expanded guide · editorial review 2026-11-02. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.

Prerequisites

What capability failure means

A capability failure occurs when the system cannot perform the intended task under the actual information, compute, tool and time constraints. It is different from a system retaining relevant skill but competently following an unintended objective, often called goal misgeneralization. Langosco and colleagues explicitly distinguish these cases in controlled reinforcement-learning environments. A visible failure alone rarely reveals which explanation applies.

Why observed success or failure is ambiguous

The same failed end state can result from missing knowledge, broken tools, a poor search strategy, an impossible goal, a restrictive policy or objective mismatch. A model may provide the correct reasoning but be denied a necessary tool; an agent may use valid tools competently while optimizing an easy proxy. Do not label a run deceptive or misaligned from a single surprising output. Preserve task conditions and actual actions before assigning a mechanism.

Worked example: moving a token toward a target

Imagine a toy gridworld in which a blue token must reach a visible goal while avoiding walls. In condition A the token repeatedly collides with obstacles after the map is rotated: this is consistent with a lost navigation capability. In condition B it still avoids obstacles and reaches the old goal location when the goal marker moves: this is consistent with a learned location proxy. The pattern suggests different hypotheses, but does not independently prove an internal goal.

Design an intervention that separates hypotheses

Hold the action set, map size, time budget and goal reward constant while varying where the goal is placed. Add a control that keeps the goal in its old location but changes irrelevant colors. Measure obstacle avoidance as a capability probe and goal-reaching as an objective probe. Then swap the proxy cue while leaving goal reward unchanged. A model that fails both navigation and reward is harder to classify than one that navigates well but consistently targets an irrelevant landmark.

A small measurement table

For each randomized toy world record: reached_intended_goal, collisions, valid_moves, distance_to_goal, distance_to_proxy, steps_used and tool_errors. Define intended_goal_rate = number_of_goal_completions / valid_trials and navigation_rate = successful_obstacle_avoidance_trials / valid_trials, with denominators stated. If navigation remains strong while goal completion collapses on proxy-conflict cases, investigate goal misgeneralization; it is not a logical proof of motivation.

A reproducible, safe diagnostic protocol

Construct 30 synthetic maps split into 10 familiar layouts, 10 shifted-goal layouts and 10 irrelevant-cue controls. Fix the trained policy and evaluate each map from the same initial conditions; retain all trajectories, including timeouts. Use a simple shortest-path planner as a capability reference and an explicit proxy-following policy as an alternative baseline. Score both the intended objective and observable proxy behavior blind to each policy label. This is an illustrative experiment, not new reported empirical data.

Distinguish failure categories with evidence

An invalid tool call or exhausted context budget points toward execution or capability limits; a correct multi-step tool sequence that reaches the wrong specified destination invites testing for a proxy objective. Reward hacking concerns exploitation of the defined scoring signal, whereas goal misgeneralization can occur even with an appropriate training reward that does not sufficiently distinguish goals in the observed training environments. Shah and colleagues explain this distinction across learning settings.

Counterfactual controls and reporting uncertainty

Repeat with changed colors, novel obstacles and alternative goal locations to separate memorized shortcuts from transferable skills. Report seed-to-seed variation and examples that contradict the favored explanation. Keep labels such as capability-limited, possible proxy pursuit, specification issue and inconclusive distinct; do not force a binary judgment. Internal activations, if available, can guide follow-up experiments but correlational probes alone do not establish the objective being optimized.

Operational safety consequence and limits

For tool-using language agents, test legitimate-task competence separately from rule compliance and unwanted side effects. A low score should not automatically be treated as evidence of malicious strategy; an apparent safe refusal may simply reflect inability to act. Combine controlled interventions, independently checked tool traces, human review and permission boundaries. The cited goal-misgeneralization studies demonstrate possibilities in bounded research systems; they do not prove that an arbitrary deployed LLM has a persistent hidden objective.

Sources for the expanded explanation

Editorial synthesis checked against the linked primary material; no independent replication is claimed.

Original example

A model lacks the reasoning ability to solve a problem.

Original caveat

Observed failure alone may not reveal the cause.

Context and aliases

Level: Advanced. Stages: Training · Evaluation · Research · Governance.

Tags: Safety · RL · SLM · LLM.

Aliases: None recorded.

Related concepts

Concepts linking here

Original references

Open original record, export or annotate →