Video Classification Basics

Video classification labels a whole clip with one tag, sampling frames smartly and pooling them so the majority verdict survives boring stretches.

Video classification samples twelve snippets and pools their votes, so celebration at 0.457 edges out play at 0.438.

Why Does This Exist?

Tagging uploads, flagging highlight reels, and routing content all need one label per clip, not per frame or per second. Dense frame classifiers drown in redundancy while missing the point. Video classification is the cheapest useful video task: sample, embed, pool, decide. It is the parent of action recognition, which names human verbs, and the entry point to video understanding, which also asks when and where.

This page covers clip-level prediction only. Per-frame backbones live in image classification; finding event boundaries needs localization instead.

Think of It Like This

Judging a parade from snapshots

You cannot watch a three-hour parade, so you photograph one float per block and judge the theme from the spread. Ten marching bands and two clowns means a music parade even if one photo caught a nap. Clip classifiers judge parades the same way: sparse snapshots, pooled verdicts, robust to the odd dull frame.

The analogy stops at the pooling. You weigh the finale more than a dull middle, while mean pooling weighs every sample equally unless an attention mechanism says otherwise.

How It Actually Works

Sampling, embedding, pooling

Uniform or segment-based sampling picks $K$ frames or short snippets spread across the clip, so a dozen samples can cover minutes of footage. A 2D or 3D backbone embeds each snippet into a vector $v_k$ . Temporal pooling fuses them into one clip vector: mean pooling takes $\bar{v} = (1/K) \sum_k v_k$ , while attention pooling learns weights $\alpha_k$ with $\sum_k \alpha_k = 1$ and uses $c = \sum_k \alpha_k v_k$ to down-weight dull stretches. A classifier turns the clip vector into probabilities, training uses video-level cross-entropy $L = -\log p_y$ for the true label $y$ , and inference averages several multi-crop views for stability.

Worked twelve-snippet vote

Twelve snippets from a soccer compilation score celebration at 0.9 on four snippets, midfield play at 0.7 on six, and crowd shots near uniform on two. The celebration mean is $(4 \times 0.9 + 6 \times 0.2 + 2 \times 0.34)/12 = 5.48/12 = 0.457$ . The play mean is $(4 \times 0.1 + 6 \times 0.7 + 2 \times 0.33)/12 = 5.26/12 = 0.438$ . Celebration wins despite fewer snippets because its peaks are stronger, and attention pooling would widen the gap by explicitly down-weighting the crowd frames.

Code

1celebration = [0.9] * 4 + [0.2] * 6 + [0.34] * 22play = [0.1] * 4 + [0.7] * 6 + [0.33] * 23print(round(sum(celebration) / 12, 3))4# -> 0.4575print(round(sum(play) / 12, 3))6# -> 0.438

Watch Out For

Single-label verdicts on multi-event clips

One tag per clip fails compilations and long untrimmed videos where several events share the runtime. The symptom is oscillating predictions across seeds as pooling ties break randomly. Switch to multi-label heads, or to temporal localization when a clip holds more than one event.

Sampling past a brief event

Sparse segment sampling can step clean over a two-second action inside a ten-minute clip. The symptom is a confident wrong label with every sampled snippet looking reasonable in isolation. Densify the snippets around motion peaks, or move to localization when brief events are the whole task.

The Quick Version

One label per clip comes from sampled snippets that are embedded, pooled, and classified.
Segment sampling stretches about a dozen frames across minutes of footage affordably.
Mean pooling weighs every snippet equally; attention pooling learns to down-weight dull ones.
The worked vote gives celebration 0.457 against play 0.438, so peak strength beats snippet count.
Multi-event videos need multi-label heads or localization, not single tags.