Optimization and training dynamics · MLD-0478

Gradient descent

Updating parameters in the direction opposite the gradient of an objective.

Expanded guide · editorial review 2026-11-01. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.

Prerequisites

How it works

Gradient descent updates parameters in the direction opposite the local gradient of an objective. The learning rate controls step size. Full-batch, stochastic and mini-batch methods differ in how much data estimates each gradient. Backpropagation computes gradients for a network; the optimizer uses those gradients to choose an update.

Worked example

For J(w) = (w - 3)^2, the derivative is 2(w - 3). Starting at w = 0 with learning rate 0.1 gives w_new = 0 - 0.1(-6) = 0.6. The objective falls from 9 to 5.76. This convex toy calculation does not establish convergence for a large neural network.

The update

theta_next = theta - learning_rate * gradient(J(theta)). Compute the gradient at the current parameters before updating them.

What to test

Check a small problem against an analytic gradient or finite differences. Track loss, gradient norms and held-out performance; inspect sensitivity to the learning rate and random seed.

Common pitfall

A falling training loss does not establish improved generalization. Excessive step sizes can also overshoot or diverge.

Sources for the expanded explanation

Editorial synthesis checked against the linked primary material; no independent replication is claimed.

Original example

Reduce a differentiable training loss.

Original caveat

A step that is too large can increase the objective.

Context and aliases

Level: Foundational. Stages: Training · Theory.

Tags: ML · DL · SLM · LLM.

Aliases: GD.

Related concepts

Concepts linking here

Original references

Open original record, export or annotate →