Math

Vectors, Matrices and Tensors

Three names for the same idea at different sizes: a container of numbers, described entirely by how many axes it has and how long each one is.

Rank is just how many axes a container has: no axes is a scalar, one is a vector, two is a matrix, and a real batch of images needs four

Why Does This Exist?

Read any ML stack trace and you will find a shape in it. expected (32, 10), got (32, 1). cannot reshape array of size 4816896 into shape (32,224,224). Shape errors are, by a wide margin, the most common bug in day-to-day machine learning work — and the dangerous ones do not raise an error at all. They silently produce an array of the wrong size that keeps flowing downstream until your loss looks strange three hours into a training run.

This page exists because every one of those bugs is a misunderstanding of one small idea: a tensor is a container, and its shape is the complete description of it. Get fluent in shapes and a whole category of debugging disappears.

The payoff is immediate and practical. (32, 224, 224, 3) should read, at a glance, as thirty-two colour images of 224 by 224 pixels, in that axis order. If it does not yet, that is what the next few minutes are for.

Think of It Like This

An office building's addressing system

A single number is a note on a desk. To find it you need no directions at all.

A vector is one corridor of offices. "Office 4" is enough — one number locates anything.

A matrix is one floor: corridors crossing corridors. Now you need two numbers, "row 3, office 4", and the order matters. Row 3 office 4 is not office 3 row 4.

The whole building is a rank-3 container: floor, row, office. A campus of buildings is rank-4. Nothing conceptually new happens as you add axes — you just need one more number to pin down a location, and you have to agree in advance what order those numbers come in.

That last sentence is the entire source of the bugs. Nothing in the building stops you reading "floor 3, row 12" as "floor 12, row 3". You would just end up somewhere unexpected, holding a perfectly valid address.

How It Actually Works

Rank is the number of axes. That is the only classification you need:

Rank 0 — a scalar. One number. A learning rate, a loss value.
Rank 1 — a vector, shape $(n,)$ . A word embedding, one row of features, a bias.
Rank 2 — a matrix, shape $(m, n)$ . A weight layer, a batch of feature rows.
Rank 3 and above — a tensor. A batch of sequences, a batch of images.

Strictly, "tensor" covers all of them, and rank 0, 1 and 2 are just the small cases. Frameworks use the general word for everything, which is why torch.Tensor holds a scalar perfectly happily.

A vector $v$ with $n$ components is written $v \in \mathbb{R}^{n}$ , and a matrix with $m$ rows and $n$ columns as:

X \in \mathbb{R}^{m \times n}

In plain words: $X$ holds real numbers arranged in $m$ rows by $n$ columns. The convention throughout ML is that the first axis is the batch axis — one entry per training example — so $X \in \mathbb{R}^{32 \times 768}$ is thirty-two examples of 768 features each, never the transpose of that.

Axes have meaning, and the meaning is not stored

This is the sentence to take away. A shape of (32, 10) tells you there are 32 of something and 10 of something else. It does not tell you which is the batch. Nothing in the array records that the first axis means "example" and the second means "class" — that agreement lives only in your head and in your code's consistency.

So every operation that reduces or reorders has to be told which axis to act on, and getting it wrong is not an error, it is a different answer. Take a small matrix:

A = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}

Summing down the rows (axis=0) collapses the first axis and gives $[5, 7, 9]$ with shape $(3,)$ . Summing across the columns (axis=1) collapses the second and gives $[6, 15]$ with shape $(2,)$ . Both are valid sums of $A$ . Only one is the per-example total you probably wanted.

Broadcasting is convenient and it is where the silent bugs live

When shapes do not match, NumPy and PyTorch do not necessarily complain. They line the shapes up from the right and stretch any axis of length 1 to match:

$(32, 10)$ with $(10,)$ → aligns to $(32, 10)$ . This is the bias-add you want.
$(32, 1)$ with $(32,)$ → aligns to $(32, 32)$ . This is almost certainly a bug.

That second case deserves staring at. A column of 32 predictions and a flat row of 32 labels do not subtract elementwise. They expand into a 32 by 32 matrix of every prediction minus every label, your loss silently becomes the mean of 1,024 numbers instead of 32, and training does something plausible-looking but wrong.

Worked example

Take one batch of colour images: 32 images, 224 by 224 pixels, 3 channels. The shape is $(32, 224, 224, 3)$ .

Count the elements by multiplying the axes:

32 \times 224 \times 224 \times 3 = 4{,}816{,}896

In plain words: that shape holds a little under five million individual numbers. At float32, four bytes each, that is $4{,}816{,}896 \times 4 = 19{,}267{,}584$ bytes — exactly 18.375 MiB for a single batch.

That number is why batch size is a memory decision. Double the batch and you double it; switch to float16 and you halve it. And remember this is only the input — every intermediate activation in the forward pass has to be held too, because the chain rule needs those values on the way back.

Show Me the Code

1import numpy as np2
3batch = np.zeros((32, 224, 224, 3), dtype=np.float32)  # (batch, H, W, channels)4print(batch.ndim, batch.size)          # -> 4 48168965print(batch.nbytes / 1024**2)          # -> 18.375  (MiB, float32)6
7A = np.array([[1, 2, 3], [4, 5, 6]])   # shape (2, 3)8print(A.sum(axis=0), A.sum(axis=0).shape)  # -> [5 7 9] (3,)   collapses rows9print(A.sum(axis=1), A.sum(axis=1).shape)  # -> [ 6 15] (2,)   collapses columns10
11# The broadcasting trap: neither line errors, and only one is what you meant.12y_true, y_pred = np.zeros(32), np.zeros((32, 1))13print((y_pred - y_true).shape)             # -> (32, 32)  silently wrong14print((y_pred.ravel() - y_true).shape)     # -> (32,)     what you wanted

The last two lines are worth committing to memory. That (32, 32) is one of the most expensive four characters in machine learning.

Watch Out For

Using reshape to reorder axes

reshape and transpose both change a shape, and they are not interchangeable. transpose moves axes and carries the data with them. reshape keeps the data in memory order and re-cuts it into a new shape.

So converting a channels-last batch $(32, 224, 224, 3)$ to channels-first $(32, 3, 224, 224)$ with reshape succeeds. The element count matches, no error is raised, and the shape afterwards is exactly what you asked for. But the pixels have been interleaved into nonsense — channel values from neighbouring pixels are now sitting where a whole channel plane should be.

The model trains. It just learns from scrambled images and plateaus at an accuracy you then go looking for architectural reasons to explain. Reshape when you are splitting or merging adjacent axes; transpose when you are reordering them. If you cannot say which you are doing, you are about to write this bug.

Trusting a shape that only looks right

A correct shape is weak evidence. (32, 32) from the broadcasting example above is a perfectly reasonable-looking shape, and so is the $(3, 3)$ you get from multiplying two matrices in the wrong order.

The habit that actually prevents this: assert the shapes you expect, at the boundaries of your code, in plain assertions. assert logits.shape == (batch, n_classes) costs nothing and converts a silent wrong answer into an immediate failure at the exact line responsible. Frameworks will not do this for you, because they cannot know what your axes mean.

A useful sharpening: after any reduction, check the rank dropped by exactly as much as you intended. A stray keepdims=True, or a missing one, is the difference between $(32,)$ and $(32, 1)$ — and that difference is what feeds the broadcasting trap on the next line.

The Quick Version

Rank is the number of axes. Scalar 0, vector 1, matrix 2, batch of images 4.
Shape fully describes the container: (32, 224, 224, 3) is 32 images, 224 by 224, 3 channels.
The first axis is the batch axis by convention. That convention is not stored anywhere.
Axes carry meaning your code assigns, so every reduction must be told which axis to act on.
Broadcasting aligns shapes from the right and stretches length-1 axes. (32, 1) against (32,) quietly becomes (32, 32).
Multiply the axes for the element count; times the dtype size for the memory. One float32 image batch here is 18.375 MiB.
transpose reorders axes and moves data. reshape re-cuts memory order. Confusing them scrambles data without erroring.