RetinaNet Detector

Why Does This Exist?

In 2017 one-stage detectors trailed two-stage accuracy by several points, and Lin and colleagues diagnosed why: a dense grid produces around 100,000 candidates per image with only a handful of foreground, so easy background boxes dominate the loss. The network spends training getting better at background it already knows.

RetinaNet kept one-stage speed and fixed the learning dynamics with focal loss, matching Faster R-CNN accuracy while staying single-shot. It proved the gap was a loss problem, not an architecture problem.

This page covers the pyramid backbone, the two heads, and the loss math. The loss itself gets its own treatment on the focal loss page.

Think of It Like This

How It Actually Works

Backbone plus two subnetworks

A ResNet with a feature pyramid supplies multi-scale maps P3 through P7, so small and large objects each get a level that fits. Two small fully convolutional subnetworks attach to every level: one predicts class scores per anchor (3 scales $\times$ 3 aspect ratios each spot), the other predicts box offsets. Both heads share weights across all pyramid levels, keeping the parameter count flat while coverage multiplies.

Focal loss mechanics

Focal loss multiplies cross-entropy by $(1 - p_t)^\gamma$ with $\gamma = 2$: $FL(p_t) = -(1 - p_t)^\gamma \log(p_t)$, where $p_t$ is the model's probability on the true class and $\log$ is the natural logarithm. When the model is confident and right, $(1 - p_t)$ is tiny and the example nearly vanishes from the gradient. When it struggles, the factor stays near $1$ and the example counts almost fully.

Worked example: easy versus hard

Score two candidates with $\gamma = 2$:

• Easy background at $p_t = 0.9$: cross-entropy $-\log(0.9) = 0.1054$, focal loss $(0.1)^2 \times 0.1054 = 0.0011$. Down about $100$ times.
• Hard case at $p_t = 0.3$: cross-entropy $-\log(0.3) = 1.2040$, focal loss $(0.7)^2 \times 1.2040 = 0.5899$. Down only about $2$ times.

Easy examples nearly leave the gradient while hard ones survive almost intact. That asymmetry is the whole trick: the rare foreground finally steers training instead of drowning in background.

Code

1import math2
3def focal_loss(pt, gamma=2.0):4    ce = -math.log(pt)  # cross-entropy for the true class5    return ((1.0 - pt) ** gamma) * ce6
7for pt in (0.9, 0.3):8    print(pt, f"{-math.log(pt):.4f}", f"{focal_loss(pt):.4f}")9# -> 0.9 0.1054 0.001110# -> 0.3 1.2040 0.5899

Watch Out For

The Quick Version

• RetinaNet combines a feature pyramid with focal loss in a one-stage design.
• Focal loss scales cross-entropy by $(1 - p_t)^2$, erasing easy background from the gradient.
• Verified effect: about $100$ times smaller loss on easy cases, only about $2$ times on hard ones.
• Shared heads across pyramid levels P3 to P7 keep parameters flat as coverage multiplies.
• It matched two-stage accuracy in 2017 while keeping single-shot speed.

What to Read Next

• Anchor Boxes
• Pyramids Detect Every Scale Once
• Focal Loss Function