Faster R-CNN Learns Its Own Proposals

Faster R-CNN replaces slow selective search with a Region Proposal Network that learns proposals on shared GPU features, unifying the whole detector into one network.

Faster R-CNN slides a Region Proposal Network over shared features to propose regions, then classifies each with a second-stage head, turning anchor 150, 150, 50, 100 into proposal 155, 130, 50, 110.52

Why Does This Exist?

Fast R-CNN was fast except for selective search, which still burned about 2 seconds per image on CPU proposing regions. Ren and colleagues saw in 2015 that proposals can be learned by a small network sliding over the same GPU feature map the detector already computes. The Region Proposal Network costs almost nothing, runs in milliseconds, and the full system hits 5 frames per second while topping accuracy charts. It became the reference two-stage detector for a decade, and the general detection setup it refines lives in object detection.

Think of It Like This

A scout who reads the same map

Fast R-CNN hired scouts who walked the terrain on foot while appraisers studied aerial photos. Faster R-CNN teaches the scouts to read the same aerial photo: they scan the shared map, circle promising blocks, and hand the marked map to the appraisers.

One map, two readers, no footwork. Proposals and classification finally share both the features and the GPU. Where it stops: scouts circling blocks still cost a second reading pass, which is why one-stage detectors later skipped the circling step entirely for raw speed.

How It Actually Works

The RPN head

A 3 by 3 convolution slides over the backbone feature map. At each position, $k$ anchor boxes of different scales and aspect ratios, usually 9, each get an objectness score and four box offsets $(t_x, t_y, t_w, t_h)$ . Anchors overlapping a ground-truth box above IoU 0.7 train as positive, below 0.3 as background, and the rest are ignored. Top proposals pass through non-maximum suppression down to about 300 survivors, which is where non-maximum suppression earns its place in the pipeline.

The second stage

Survivors are pooled with RoI pooling into fixed 7 by 7 crops, then split into a classifier over $C + 1$ classes and a box regressor. The multitask loss adds the RPN classification and regression terms, so one backward pass trains proposals and decisions together. Paired with a Feature Pyramid Network backbone, small-object recall jumps because each pyramid level proposes at its own scale. The Mask R-CNN extension later added a mask branch on the same two-stage skeleton.

Worked example

Decode one anchor into a proposal by hand. The anchor is $(x, y, w, h) = (150, 150, 50, 100)$ and the network predicts offsets $(t_x, t_y, t_w, t_h) = (0.1, -0.2, 0.0, 0.1)$ . The center shifts to $150 + 0.1 \times 50 = 155$ and $150 + (-0.2) \times 100 = 130$ . The size scales to $50 \times e^{0.0} = 50$ and $100 \times e^{0.1} \approx 100 \times 1.1052 = 110.52$ . The proposal is $(155, 130, 50, 110.52)$ : nudged right, pulled up, slightly taller.

Code

The snippet below performs that anchor decoding with only the standard library, using the worked example numbers.

1import math2
3anchor = (150.0, 150.0, 50.0, 100.0)  # x_center, y_center, w, h4tx, ty, tw, th = (0.1, -0.2, 0.0, 0.1)5
6px = anchor[0] + tx * anchor[2]  # 150 + 5 = 1557py = anchor[1] + ty * anchor[3]  # 150 - 20 = 1308pw = anchor[2] * math.exp(tw)    # 50 x 1 = 509ph = anchor[3] * math.exp(th)    # 100 x 1.1052 = 110.5210
11print((px, py, round(pw, 2), round(ph, 2)))12# -> (155.0, 130.0, 50.0, 110.52)

Every RPN prediction runs this arithmetic: shift the center by the offset times the anchor size, scale the size by the exponentiated log offset. Thousands of anchors decode the same way before NMS trims them to 300.

Watch Out For

Default anchors that ignore your object sizes

COCO-sized anchors miss tiny satellite ships and giant close-up defects alike. The symptom is confident detections on medium objects with vanished extremes, even though training loss looks healthy. Fix it by clustering your own box sizes with k-means before training and setting anchor scales to match, rather than retraining longer on the wrong grid.

Expecting video-rate speed

Five frames per second is research real-time, not deployment real-time. The symptom is a demo that stutters on live video while accuracy looks excellent offline. Fix it by switching families to a one-stage detector for 30-plus frames per second instead of pruning a two-stage one and losing the accuracy you chose it for.

The Quick Version

Faster R-CNN learns proposals with an RPN sliding over shared backbone features.
Nine anchors per position get objectness scores plus box offsets, trimmed by NMS to about 300.
The worked anchor decodes to proposal (155, 130, 50, 110.52) through shift and scale arithmetic.
RoI pooling plus a joint multitask loss trains proposals and heads in one network.
It leads on accuracy, especially small objects with FPN, at about 5 frames per second.