Optimisation

Why this exists

The one idea

Walking downhill, by hand

Let's make it concrete with the smallest possible example. Our cost function — how wrong we are as a function of the knob — is a parabola:

The slope of that curve at any point is dJ/dw = 2(w − 3). Start at w = 8 with a learning rate of 0.1 and turn the handle three times:

• At w = 8 the slope is 2 × 5 = 10. Positive slope means the ground rises to the right, so go left: w ← 8 − 0.1 × 10 = 7.0
• At w = 7 the slope is 2 × 4 = 8. Step of 0.8: w ← 6.2
• At w = 6.2 the slope is 6.4. Step of 0.64: w ← 5.56

Look at what happened to the step sizes: 1.0, then 0.8, then 0.64. Nobody told it to slow down. As the curve flattens near the bottom, the slope shrinks, and the steps shrink with it — automatically. That self-braking behaviour is why this simple rule works at all.

Now the important part. Drag the learning rate up. At α = 0.5 it lands exactly on the minimum in a single step. At α = 1.0 it bounces between 8 and −2 forever, never getting closer. Above that, each bounce is bigger than the last and the whole thing flies apart. Same algorithm, same curve, same starting point — one number decides whether it works.

Why this curve is the easy case

That parabola is convex: pick any two points on it, draw a straight line between them, and the line never dips below the curve. Convexity buys you something enormous — there is exactly one minimum, and any path that keeps going downhill must eventually reach it. No starting point is unlucky. Linear and logistic regression both have convex cost functions, which is why they train reliably and why they're the right place to learn this.

Most interesting models are not convex. A neural network's cost surface is a landscape of ridges, valleys and plateaus, and where you end up depends on where you started. It also contains saddle points — places that slope down in one direction and up in another, so the gradient is zero without being a minimum at all. Think of a mountain pass: flat underfoot, but you're nowhere near the bottom. Plain gradient descent stalls there, because a zero gradient means a zero step. That single failure mode is the reason momentum and adaptive optimizers exist.

Here's the identical update rule doing something real: fitting a straight line to eight points. Two knobs this time, a slope w and an intercept b. The red bars are the residuals — the errors being squared and summed — and the inset tracks the cost falling with every iteration.

The update rule

Everything above is one line:

Plugging the very first step in: w := 8 − 0.1 × 10 = 7. That's the whole algorithm. Every model in this track, and every neural network you will ever train, is that line in a loop.

With more than one parameter, nothing changes conceptually — you just do it to each one. Collect the partial derivatives into a vector and you have the gradient, written ∇J:

For the line-fitting example, the cost is mean squared error and the two partials work out to:

Twelve lines of Python

Same curve, same starting point, same learning rate as the first visualization. Run it and compare the numbers to what you saw on the chart.

def J(w):
    return (w - 3) ** 2 + 1

def grad(w):
    return 2 * (w - 3)          # dJ/dw, worked out by hand

w = 8.0                         # where we happen to start
lr = 0.1                        # the learning rate, alpha

for step in range(5):
    g = grad(w)
    w = w - lr * g              # THE update rule. This is all of it.
    print(f"step {step + 1}:  gradient={g:6.2f}   w={w:6.3f}   J(w)={J(w):7.4f}")

Now the two-parameter version, fitting the same eight points as the second visualization. Note there is still no library doing anything clever — NumPy is here only so that w * X multiplies the whole array at once instead of needing a loop.

import numpy as np

X = np.array([1, 2, 3, 4, 5, 6, 7, 8], dtype=float)
y = np.array([2.1, 3.9, 6.2, 7.8, 10.1, 12.2, 13.8, 16.1])
n = len(X)

w, b = 0.0, 0.0                 # start with a flat line through the origin
lr = 0.01

for i in range(1, 5001):
    err = (w * X + b) - y       # how wrong each prediction is
    gw = (2 / n) * np.sum(err * X)
    gb = (2 / n) * np.sum(err)
    w -= lr * gw                # same update rule, once per parameter
    b -= lr * gb
    if i in (1, 10, 100, 1000, 5000):
        mse = np.mean(((w * X + b) - y) ** 2)
        print(f"iter {i:>4}:  w={w:.4f}  b={b:.4f}  MSE={mse:.5f}")

The exact least-squares answer for this data is w = 1.9976, b = 0.0357, with an MSE of 0.02435. Gradient descent found it to four decimal places without ever solving an equation — just by feeling the slope five thousand times.

One detail worth noticing: after 10 iterations w was already at 1.94, but b took a thousand more to settle. The two parameters converge at wildly different speeds because x ranges up to 8 while the intercept's "feature" is always 1. That's not a quirk of this dataset — it's the reason feature scaling exists.

What you'd actually write

Nobody hand-derives gradients at work. In scikit-learn the whole loop is one fit call:

from sklearn.linear_model import SGDRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

# StandardScaler is not optional here — see the pitfalls below.
model = make_pipeline(
    StandardScaler(),
    SGDRegressor(
        loss="squared_error",
        learning_rate="invscaling",   # alpha shrinks as training goes on
        eta0=0.01,                    # the starting learning rate
        max_iter=5000,
        tol=1e-6,                     # stop early once improvement stalls
    ),
)
model.fit(X.reshape(-1, 1), y)

And in PyTorch, where you'd be training something with rather more than two parameters, you never write the derivative at all — loss.backward() computes every partial derivative in the whole network by applying the chain rule automatically:

import torch

model = torch.nn.Linear(1, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
loss_fn = torch.nn.MSELoss()

for epoch in range(5000):
    optimizer.zero_grad()             # clear last step's gradients
    pred = model(X_tensor)
    loss = loss_fn(pred, y_tensor)
    loss.backward()                   # fills .grad on every parameter
    optimizer.step()                  # w -= lr * w.grad, for every w

What the framework gives you that your version didn't: automatic differentiation, so you never derive a gradient by hand again; numerically stable reductions; and vectorised, GPU-parallel arithmetic.

Momentum, in one idea

It also gives you better update rules. The most important one is momentum, and the intuition is exactly what the word suggests: instead of stepping purely by the current slope, keep a running velocity and let each gradient nudge it.

Two things fall out of that. Across a plateau or a saddle point, where the gradient is near zero, the accumulated velocity carries you through instead of stalling. And down a long narrow ravine — the unscaled-features case — the side-to-side oscillations point in opposite directions on alternating steps and cancel out in the average, while the consistent downhill component adds up. You get less zigzag and faster progress at once.

Adam, the default choice for most deep learning, combines momentum with a separate adaptive learning rate per parameter — which fixes exactly the slow-b problem you saw above. In PyTorch it's a one-word change: torch.optim.Adam(model.parameters(), lr=0.01).

When to reach for it

That last point deserves emphasis, because it sounds fatal and isn't. On a landscape with many valleys, gradient descent settles into whichever one it happened to start above. For deep networks this turns out not to matter much in practice — in very high dimensions, most stationary points are saddle points rather than bad local minima, and the minima that exist tend to be roughly as good as each other.

What goes wrong, and the fix

Interview questions & what's next