Why gradient descent zigzags

Gradient descent on a quadratic is the “hello world” of optimization, and it already contains the most important idea in the subject: the shape of the loss matters more than its size. Here is the whole argument, small enough to fit on one whiteboard.

The setup

Take a quadratic $f(x) = \tfrac{1}{2} x^\top A x$ with $A$ symmetric positive definite, and call its smallest and largest eigenvalues $\mu$ and $L$. The gradient is $\nabla f(x) = Ax$, so a step of gradient descent with step size $\eta$ is

$$x_{k+1} = x_k - \eta \nabla f(x_k) = (I - \eta A)\, x_k. \tag{1}$$

Now write $x$ in the eigenbasis of $A$.This is the whole trick. In the eigenbasis the problem splits into independent one-dimensional problems, one per eigenvalue, and each one is trivial. Each coordinate evolves on its own, shrinking by a fixed factor at every step:

$$x^{(i)}_k = (1 - \eta \lambda_i)^k \, x^{(i)}_0 .$$

So we converge only if $|1 - \eta \lambda_i| < 1$ for every eigenvalue, which means $\eta < 2/L$.Push $\eta$ past $2/L$ in the toy below and watch the red path leave the board. The slowest coordinate sets the pace, and the best we can do is to balance the two extremes with $\eta^\star = 2/(L + \mu)$, giving the rate

The condition number $\kappa$ is everything. When $\kappa = 1$ the contours are circles, the negative gradient points straight at the minimum, and a single step of size $1/L$ lands on it. When $\kappa$ is large the contours are long, thin ellipses. The gradient is perpendicular to the contour, not aimed at the minimum, so it mostly points across the valley: we bounce between the walls while creeping along the floor.

Play with it

The toy below runs both methods on a quadratic with $\kappa = 16$. Click anywhere on the board to restart from that point.

step size η =

momentum β =

↻ replay

gradient descent

momentum

Momentum

Polyak’s heavy-ball method adds a fraction of the previous step:think: a ball rolling in the bowl, not a hiker taking careful steps

$$x_{k+1} = x_k - \eta \nabla f(x_k) + \beta \,(x_k - x_{k-1}).$$

With $\eta$ and $\beta$ tuned to the curvature, the rate improves from $(\kappa - 1)/(\kappa + 1)$ to $(\sqrt{\kappa} - 1)/(\sqrt{\kappa} + 1)$.Polyak’s tuning: $\eta = 4/(\sqrt{L} + \sqrt{\mu})^2$ and $\beta = \rho^2$, where $\rho$ is that new rate. That square root is not a small detail. With $\kappa = 100$:

method	rate per step	steps to shrink the error $10^6\times$
gradient descent	$99/101 \approx 0.980$	$\approx 690$
heavy ball	$9/11 \approx 0.818$	$\approx 69$

A tenfold speed-up for one extra vector of memory. The whole thing is a few lines:

import numpy as np

def heavy_ball(A, x0, lr, beta, steps):
    x, prev = x0, x0
    for _ in range(steps):
        x, prev = x - lr * (A @ x) + beta * (x - prev), x
    return x

The takeaway

Much of what makes modern optimizers work, including momentum, preconditioning and Adam’s per-coordinate step sizes, can be read as an attempt to make $\kappa$ smaller, or to stop caring about it. The zigzag is what it looks like when you don’t.

@misc{prakash2026gradient,
  author = {Arya Prakash},
  title  = {Why gradient descent zigzags},
  year   = {2026},
  month  = {oct},
  url    = {https://jirachi.ai/posts/why-gradient-descent-zigzags/},
}