Math

Derivatives and Partial Derivatives

A single number that answers one question: if I nudge this input a little, how much does the output move, and in which direction?

The derivative at a point is the slope of the line that just touches the curve there, measured as rise over run; a partial derivative is the same slope taken while every other input is held still

Why Does This Exist?

A model learns by changing numbers. To change one sensibly you need to know what changing it does — and "the loss went from 2.31 to 2.29" is useless on its own, because you cannot tell whether that was your nudge or the noise in the batch. What you need is the response per unit of change, isolated to the one number you touched.

That quantity is the derivative, and it is the atom of every training algorithm in use today. Learning rate schedules, Adam, weight decay, gradient clipping, LoRA — all of them are strategies for spending derivatives well. If this page is fuzzy, everything built on it stays fuzzy, which is why it is the first page of the band.

There is a second reason it matters practically. A derivative tells you not just the direction but the sensitivity. When one parameter has a derivative of 0.0001 and another has 400, you are looking at the reason a single global learning rate struggles, and the reason adaptive optimisers were invented.

Think of It Like This

The speedometer, not the odometer

Your odometer says you have driven 180 kilometres. That is the total. It tells you nothing about what is happening right now — you could be parked.

The speedometer is the derivative. It reads the rate at this instant: kilometres per hour, right here. Press the accelerator and it climbs immediately, long before the odometer notices.

Everything about derivatives is in that distinction. The odometer is the function; the speedometer is its derivative. And notice the speedometer is a local reading — it tells you about now, and says nothing about the traffic jam in ten minutes. Derivatives are local too, which is exactly why training takes many small steps instead of one confident leap.

How It Actually Works

Start with the honest definition. The derivative of $f$ at a point $x$ is what the rise-over-run slope settles down to as you shrink the run to nothing:

f'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h}

In plain words: move a tiny step $h$ to the right, see how much the output changed, and divide by the size of the step. The limit says "and keep shrinking that step until the answer stops changing."

That is a definition, not a method. In practice you use rules, and for machine learning you need surprisingly few. The power rule, $\frac{d}{dx}x^n = nx^{n-1}$ , covers polynomials. Sums differentiate term by term. Constants differentiate to zero. That is enough for this page; products and compositions are the chain rule's territory.

Reading the sign and the size

Two pieces of information come out, and they get used differently:

The sign says which way. Positive means the output rises as the input rises, so to reduce the output you decrease the input. This is where training's minus sign comes from.
The size says how much, per unit. A derivative of 400 means a nudge of 0.001 moves the output by roughly 0.4. A derivative of 0.0001 means that same nudge barely registers.

A derivative of exactly zero is a flat spot — a stationary point. It might be a minimum, a maximum, or a saddle, and the derivative alone cannot tell you which.

Partial derivatives: the same idea, more knobs

Real models have millions of inputs, not one. The extension is almost anticlimactic: differentiate with respect to one variable and treat every other variable as a constant. Nothing else changes.

Write it with a curly $\partial$ instead of a straight $d$ to signal that other variables exist and are being held still:

\frac{\partial f}{\partial x}

Read as "the change in $f$ per unit of $x$ , with everything else frozen." The freezing is the whole trick, and it is why the mechanical work is no harder than the single-variable case — a term with no $x$ in it is a constant, and constants differentiate to zero.

Collect one partial derivative per parameter into a single vector and you have a gradient, which is the object optimisers actually consume.

Worked example

Take $f(x) = x^2$ and stand at $x = 2$ .

By the power rule, $f'(x) = 2x$ , so $f'(2) = 4$ . The output climbs four times as fast as the input, right there.

You can see that number on the diagram above as a slope triangle. Step 0.5 to the right along the tangent and it rises by 2, so the slope is $2 \div 0.5 = 4$ . Note that the slope is not a property of the curve as a whole — at $x = 1$ it would be 2, and at $x = 0$ it would be 0. Local, always.

Now the two-variable case. Take $g(x, y) = x^2 + 3y^2$ at the point $(2, 1)$ :

\frac{\partial g}{\partial x} = 2x = 4, \qquad \frac{\partial g}{\partial y} = 6y = 6

For the first, $3y^2$ contained no $x$ , so it was a constant and vanished. For the second, $x^2$ was the constant. The output is more sensitive to $y$ here than to $x$ , even though we are further along $x$ — that asymmetry comes from the coefficient 3, and it is exactly the kind of imbalance that makes optimisation harder.

Show Me the Code

1import numpy as np2
3
4def g(x: float, y: float) -> float:5    return x**2 + 3 * y**26
7
8def partial(f, x: float, y: float, axis: int, h: float = 1e-5) -> float:9    # Perturb ONE input, hold the other still — that is the whole definition.10    dx, dy = (h, 0.0) if axis == 0 else (0.0, h)11    return (f(x + dx, y + dy) - f(x - dx, y - dy)) / (2 * h)12
13
14print(partial(g, 2.0, 1.0, axis=0))  # -> 4.000000... (analytic 2x = 4)15print(partial(g, 2.0, 1.0, axis=1))  # -> 6.000000... (analytic 6y = 6)16print(np.round([partial(g, 2.0, 1.0, axis=a) for a in (0, 1)], 6))  # -> [4. 6.]

The axis argument is the "hold everything else still" instruction, written out. Both numbers match the ones derived by hand, which is the point of running it.

Watch Out For

Reading a derivative as a prediction instead of a rate

A derivative of 4 does not mean "the output will increase by 4." It means "the output increases by about 4 units per unit of input, for a small enough step."

That word small is load-bearing, and misreading it is the most common practical error. With $f(x) = x^2$ at $x = 2$ , the derivative predicts that stepping 1 to the right raises the output by 4, giving 8. The true value is $f(3) = 9$ . The derivative was not wrong — the step was too big for a straight-line approximation of a curve to hold.

This is precisely why training uses a learning rate to take small steps rather than jumping to where the slope says the bottom is. A large learning rate is a bet that the local slope holds over a long distance, and on a curved surface that bet loses.

Differentiating at a point where the derivative does not exist

Not every function has a derivative everywhere, and ML is full of the exceptions. ReLU has a corner at zero. abs() has one too. The L1 norm has corners along every axis. At a corner the slope from the left and the slope from the right disagree, so there is no single answer.

Frameworks do not error on this. They pick a value — often 0 or 1 — and carry on, so the failure is silent. Usually it is harmless, because landing exactly on the corner in floating point is rare and the subgradient chosen is a legitimate one.

Where it bites is when your data makes the corner common rather than rare. Feed a batch of exact zeros through ReLU, or apply L1 to a tensor that is already sparse, and a large fraction of your gradients come from an arbitrary convention rather than from the mathematics. If a model plateaus with suspiciously many exact zeros in its gradients, this is worth checking before anything else.

The Quick Version

A derivative is a rate of change: output movement per unit of input movement, measured at one point.
Its sign gives the direction and its size gives the sensitivity. Training uses both.
It is local. It describes the slope right here and promises nothing further away, which is why steps are small.
A partial derivative is the identical idea with other variables held frozen — written $\partial$ rather than $d$ .
Terms without the variable you are differentiating are constants, so they drop to zero. That is most of the work.
One partial derivative per parameter, collected into a vector, is a gradient.
Corners (ReLU, abs, L1) have no derivative. Frameworks silently substitute one.