RAFT Recurrent Flow Refinement

Look up motion, fix a little, repeat. RAFT keeps one high-resolution flow field and polishes it with a learned recurrent step.

RAFT looks up match costs around one flow field and applies a learned GRU update, so error falls with each repeated step.

Why Does This Exist?

Coarse-to-fine pyramids freeze their early mistakes: a wrong coarse match warps every finer level after it. Single-shot networks cannot revise at all. RAFT (Teed and Deng, ECCV 2020) exists to refine instead: build one all-pairs correlation volume at high resolution, then apply a small learned update operator to a single flow estimate again and again. Accuracy rises with the iteration count while the weights stay shared, so the model stays compact. This page covers the volume, the recurrent update, and the loss schedule that prices late precision. The general problem lives in optical flow, and the classical baseline it replaced is Lucas-Kanade.

Think of It Like This

Tracing a portrait with repeated glances

You sketch a face roughly, glance at the model, fix the nose, glance again, fix the eyes, and repeat until the likeness locks. Each glance is cheap and the drawing stays one sheet: you never restart on fresh paper.

The flow field is the sketch and correlation lookups are the glances. It stops at the stopping rule: you quit by feel, while RAFT runs a fixed dozen steps whose training loss weights late steps far above early ones.

How It Actually Works

Two encoders turn the frame pair into feature maps at $1/8$ resolution, where $f_1$ and $f_2$ are the feature grids of frames one and two. Every vector in $f_1$ is dotted with every vector in $f_2$ to form a 4D all-pairs correlation volume, pooled into four scales so each lookup sees both fine and wide context. Each iteration indexes that volume around the current flow estimate $f_k$ , feeds the retrieved costs plus flow and context features $x_k$ into a convolutional GRU with hidden state $h_k$ , and adds the predicted correction: $h_{k+1} = \text{GRU}(h_k, x_k)$ , $\Delta f = \text{head}(h_{k+1})$ , $f_{k+1} = f_k + \Delta f$ .

Worked numbers: the loss schedule

Training runs $N = 12$ steps with weight $w_i = \gamma^{N-i}$ on step $i$ and decay $\gamma = 0.8$ , so later steps pay more. The first step weighs $0.8^{11} \approx 0.0859$ while the last weighs $0.8^{0} = 1.0$ , a ratio of $1 / 0.0859 \approx 11.64$ : the final correction counts nearly twelve times the first guess, which frees early steps to explore and punishes late sloppiness. On a fast Sintel clip the error falls from $8.5$ px at zero flow to $2.7$ px after 4 steps and about $1.43$ px after 12, with a 24th step reaching only $1.38$ .

The volume size

For a $512 \times 640$ frame pair the $1/8$ features are $64 \times 80$ , so the all-pairs volume holds $5120 \times 5120 = 26{,}214{,}400$ correlations, about 26.2 million, pooled cheaply across scales. That quadratic growth is the method's bill, and you'll feel it the moment full-resolution video arrives: tile the frames or downsample before the volume, and cost that step in your latency budget.

Code

1def raft_weights(gamma: float = 0.8, steps: int = 12) -> list:2    """Per-step training weights w_i = gamma^(N-i): late steps pay more."""3    return [round(gamma ** (steps - i), 4) for i in range(1, steps + 1)]4
5
6w = raft_weights()7print(w[0], w[-2], w[-1])8# -> 0.0859 0.8 1.09print(round(sum(w), 4))10# -> 4.6564

The first step weighs 0.0859 against the last step's 1.0, and the twelve weights sum to 4.6564, replaying the schedule in the worked example.

Watch Out For

Running too few steps on fast motion

Two iterations leave large shifts half-solved. The symptom is clean slow regions beside smeared fast cars, and judging RAFT by a single pass makes it look broken. Budget twelve steps for benchmarks and at least six for real time: step count is the accuracy knob at test time.

Out of memory on high-resolution video

The all-pairs volume grows with the square of the pixel count, so a 4K frame pair explodes it past GPU memory. The symptom is an OOM kill on frames your pyramid baseline handled fine. Tile the frames or downsample before the volume, and cost that step in your latency budget.

The Quick Version

RAFT keeps one flow field and refines it with a shared recurrent update.
An all-pairs correlation volume supplies global match evidence upfront.
A convolutional GRU proposes each additive correction from volume lookups.
Training weights the last step about 11.6 times the first with decay 0.8.
Error falls from 8.5 px to near 1.4 px over twelve steps on fast clips.
Step count sets test-time accuracy, and the volume sets the memory bill.