Interpretability and mechanistic analysis · MLD-0718

Mechanistic interpretability

Studying internal computations to identify causal mechanisms underlying model behavior.

Expanded guide · editorial review 2026-11-02. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.

Prerequisites

What mechanistic interpretability claims to explain

Mechanistic interpretability studies which internal computations cause a model to produce a particular behavior. It asks for a model-specific causal account—components, information flow and interventions that change predictions—rather than just a persuasive natural-language description. Reading an attention heatmap or a model-generated explanation is a useful hypothesis generator, not by itself proof of mechanism.

From residual streams to candidate circuits

A transformer moves representations through a residual stream while attention and MLP blocks contribute updates. A useful first approximation is residual_next = residual_now + block_output; the exact ordering depends on the architecture. Researchers can decompose the output of an attention head into a pattern of which positions it reads and what value information it writes. Candidate circuits are collections of components whose combined effect predicts a measurable behavior, not an assumption that each neuron has one human-readable purpose.

Worked example: resolving who received an object

For a small open-weight language model, compare synthetic prompts such as 'Mira handed the map to Theo. The person with the map is' against matched prompts where the recipient names are swapped. Record the margin logit(correct recipient) - logit(incorrect recipient) and verify the expected difference on a held-out set. Check alternative templates and name frequencies. The published indirect-object-identification circuit work investigated a related task in GPT-2 small; this example is a new illustration, not a claim to reproduce its numerical result.

Causal testing with activation patching

Run a clean prompt and a minimally changed corrupted prompt while saving the model's internal activations. Replace one selected corrupted activation with the corresponding clean activation, then recompute the output margin. In pseudocode: delta = margin(run(corrupt, patch=clean_head_h)) - margin(run(corrupt)). Compare this effect to random-head patches, token-alignment controls, and unpatched baselines. A large delta supports causal influence for this contrast; it does not prove that the patched head alone implements the complete algorithm.

A bounded reproducible investigation

Freeze one checkpoint and tokenizer; choose 40 clean/corrupted training pairs to formulate hypotheses and a separate 40-pair held-out test with distinct names/templates. Record baseline margins, head-level patch effects, multiple random seeds where stochasticity applies, and results for shuffled or unrelated activations. Pre-register an expected direction for the margin and inspect failed pairs rather than selecting only pretty circuit diagrams. Work on local test strings and a small model; do not collect private prompts or infer hidden motives from a visualization.

Sparse features and superposition

Individual activations can mix several features, so a single neuron may be a poor semantic unit. Sparse autoencoders learn a reconstruction of model activations using a larger set of sparsely active features, offering candidate directions to inspect. Reconstruction loss, sparsity, feature splitting and sensitivity to training choices matter: an SAE feature with an appealing label is not necessarily the model's unique or natural causal concept. Test candidate features with interventions and out-of-distribution examples.

Criteria for a convincing circuit

Evaluate faithfulness (does the proposed circuit reproduce target behavior?), completeness (what relevant effects are omitted?), and minimality (can components be removed without losing the explanation?). Measure interventions on a reserved test set and report uncertainty, seed/template sensitivity and failure cases. Positive and negative causal controls help distinguish real effects from activation-patching artifacts. The IOI circuit study explicitly used these criteria and still described gaps; a diagram is not the final validation.

Useful applications and limits

Mechanistic work can generate hypotheses about factual recall, prompt effects, safety-relevant representations and unexpected model behavior, especially when paired with behavioral evaluations. It remains labor-intensive and often local to a task, model, layer or prompt distribution. Attention weights alone are not explanations, correlations are not causal mechanisms, and identified components do not establish a model's enduring goal or certify that it is safe across unseen situations.

Sources for the expanded explanation

Editorial synthesis checked against the linked primary material; no independent replication is claimed.

Original example

Analyze attention heads and feature circuits.

Original caveat

A partial mechanism does not explain the whole model.

Context and aliases

Level: Advanced. Stages: Research · Evaluation.

Tags: ML · DL · SLM · LLM · Interpretability · Safety.

Aliases: None recorded.

Related concepts

Concepts linking here

Technology connections

Original references

Open original record, export or annotate →