Focal Loss Function

Focal loss multiplies cross-entropy by a factor that fades to zero on easy examples, so a thousand background boxes stop drowning each rare object.

Focal loss scales each example loss by how wrong the model is, so easy background boxes fade while hard objects drive learning

Why Does This Exist?

Dense detectors evaluate around 100,000 anchor boxes per image with single-digit foreground. Plain cross-entropy sums over all of them, so easy background dominates the gradient and the model converges to predicting background everywhere. Practitioners patched this with sampling heuristics until the 2017 RetinaNet paper replaced the patch with a loss: scale each example contribution by $(1 - p_t)^\gamma$ , where $p_t$ is the predicted probability on the true class. Easy examples fade automatically, with no sampling code needed, which is why RetinaNet made it the default dense detection loss.

Think of It Like This

A volume knob wired to confusion

Imagine a classroom where every student shouts answers, and the teacher attention is the gradient. Cross-entropy hands every shouter an equal microphone, so the thousand confident students chanting the obvious answer drown out the ten confused ones.

Focal loss wires each microphone to a confusion meter. Confident correct students fade to a whisper while struggling students stay loud, so the teacher hears exactly the class that needs teaching. Where it stops: a truly lost student shouting nonsense gets the loudest microphone, which is why mislabeled examples hurt more under focal loss, not less.

How It Actually Works

The formula is $FL(p_t) = -(1 - p_t)^\gamma \log(p_t)$ , with $\gamma = 2$ the standard setting and an optional $\alpha_t$ class weight. The factor $(1 - p_t)^\gamma$ is near zero when the model is confident and right, and near one when it is wrong, so the loss pays attention proportional to confusion. Setting $\gamma = 0$ recovers plain cross-entropy, and raising gamma crushes easy examples harder.

Worked example

Take $\gamma = 2$ and two examples. The easy one has $p_t = 0.9$ : cross-entropy is $-\log(0.9) = 0.1054$ , and focal loss multiplies by $(0.1)^2 = 0.01$ to give $0.0011$ , exactly 100 times smaller. The hard one has $p_t = 0.3$ : cross-entropy is $-\log(0.3) = 1.2040$ , and focal loss multiplies by $(0.7)^2 = 0.49$ to give $0.5899$ , only about 2 times smaller. Same gamma, same formula, yet the easy example loses 100 times its voice while the hard one keeps half of it.

Code

The point of this snippet is the shaping, not a training loop: plain standard-library math replays the worked example numbers so you can feel gamma before trusting it.

1import math2
3def ce(pt: float) -> float:4    return -math.log(pt)5
6def focal(pt: float, gamma: float = 2.0) -> float:7    return ((1.0 - pt) ** gamma) * ce(pt)8
9for pt in (0.9, 0.3):10    print(pt, round(ce(pt), 4), round(focal(pt), 4))11# -> 0.9 0.1054 0.001112# -> 0.3 1.204 0.5899

The easy example drops from $0.1054$ to $0.0011$ while the hard one moves only from $1.2040$ to $0.5899$ . Try $\gamma = 0$ in the same function and both rows match cross-entropy exactly, which confirms gamma is the only knob.

Watch Out For

Gamma tuned for COCO shipped on mild imbalance

Gamma 2 on a 3-to-1 imbalance over-suppresses merely medium examples and stalls learning. The symptom is a loss that flatlines high with under-confident scores everywhere. Sweep gamma from 0.5 upward on validation, and confirm plain weighted cross-entropy is not enough first.

Focal loss masking broken labels

Down-weighting easy examples concentrates learning on the hardest ones, which include mislabeled data. The symptom is training that chases noise while validation degrades late. Clean labels before sharpening focus, or cap the damage with label smoothing.

The Quick Version

Focal loss scales cross-entropy by $(1 - p_t)^\gamma$ , fading easy examples out of the gradient.
Worked effect at $\gamma = 2$ : 100 times smaller loss on the $p_t = 0.9$ case, about 2 times on the $p_t = 0.3$ case.
It replaces sampling heuristics for extreme imbalance such as 100,000-to-1 anchors.
Gamma must be tuned per imbalance level, and clean labels matter more under focus, not less.