Implicit Q-Learning

Learn strong values from logged data only. You fit the top of what you saw with expectile regression and clone the good moves with weights.

IQL fits V at 5.0 between Q values 6.0 and 4.0 with tau 0.7, targets Q at 5.05, and weights the good action 403 times more

Why Does This Exist?

Offline RL has one trap. Your data came from some old behavior policy, but your Bellman update wants to ask what would happen for a new action nobody tried. Neural nets answer with confident nonsense, and the policy chases that hallucination. That gap is distributional shift.

Earlier fixes wrestle the unseen actions directly. Batch constrained Q-learning learns a model of allowed actions. Conservative Q-learning pushes unseen values down. Both need extra sampling and tuning. Start with offline batch RL for the setup, then see this page as the in sample alternative.

This page teaches the method from Kostrikov and coauthors that never asks about unseen actions at all. You learn a value upper envelope from logged pairs only, back it into Q with plain regression, and extract the policy by weighted cloning.

The one idea you need first

A Bellman target for $(s, a, r, s')$ is $r + \gamma V(s')$ . Here $r$ is the logged reward, $\gamma$ is the discount, $V(s')$ is the value of the next state. No max over actions appears, so you never need to invent an action at $s'$ .

Think of It Like This

Picking the best logged bus route

Your city keeps logs of bus trips, with travel times for routes drivers actually drove. You cannot run test buses. You want the fastest reliable route per intersection.

You do not draw a straight line through a park and claim 30 seconds. You look only at logged roads. At each crossing you take a high percentile of logged times, say the 70th percentile good trip, not the average. Then you tell new drivers to copy the trips that beat that bar and ignore the slow ones.

Where the analogy stops: streets have hard curbs, so allowed versus forbidden is crisp. In control, allowed is a soft data cloud, so the percentile becomes a smooth expectile loss instead of a hard filter.

How It Actually Works

Three steps run together, all on dataset pairs $(s, a, r, s')$ from $\mathcal{D}$ . Symbols: $V_\psi(s)$ is the state value net, $Q_\theta(s, a)$ is the action value net, $\tau$ is the expectile level, $\beta$ is the policy temperature.

First, fit $V$ with asymmetric squared loss:

L_V(\psi) = \mathbb{E}_{(s,a) \sim \mathcal{D}}[L_2^\tau(Q(s,a) - V_\psi(s))]

with $L_2^\tau(u) = \tau u^2$ when $u \ge 0$ and $(1-\tau) u^2$ otherwise. When $\tau = 0.5$ you get the mean. When $\tau = 0.7$ or $0.9$ you pull $V$ toward the good logged actions. As $\tau$ nears 1, $V$ nears the best logged value.

Next, fit $Q$ with the state value as target:

L_Q(\theta) = \mathbb{E}[(r + \gamma V_\psi(s') - Q_\theta(s,a))^2]

The target $y = r + \gamma V(s')$ touches only the observed next state $s'$ . No candidate action at $s'$ is scored.

Finally, extract the policy with advantage $A = Q - V$ :

L_\pi(\phi) = -\mathbb{E}_{(s,a) \sim \mathcal{D}}[\exp(\beta A(s,a)) \log \pi_\phi(a \mid s)]

Good moves get copied hard, bad moves get near zero weight. No backprop through Q is needed.

Worked example

One state $s$ has two logged actions. Target Q values are $Q(s,a_1) = 6.0$ and $Q(s,a_2) = 4.0$ . Current $V(s) = 5.0$ . Set $\tau = 0.7$ , $\beta = 3.0$ , $\gamma = 0.9$ .

Expectile gaps are $u_1 = 1.0$ and $u_2 = -1.0$ :

L_2^{0.7}(1.0) = 0.7 \times 1.0 = 0.70

L_2^{0.7}(-1.0) = 0.3 \times 1.0 = 0.30

The upside error costs 2.33 times the downside, so $V$ drifts up toward 6.0.

Take transition $(s, a_1, r = 1.0, s')$ with $V(s') = 4.5$ :

y = 1.0 + 0.9 \times 4.5 = 5.05

No action was invented at $s'$ .

Advantages are $A_1 = 1.0$ and $A_2 = -1.0$ . Weights:

w_1 = \exp(3.0) \approx 20.0855

w_2 = \exp(-3.0) \approx 0.0498

Ratio $20.0855 / 0.0498 \approx 403.3$ . The actor copies $a_1$ about four hundred times harder than $a_2$ , which is the filtering the diagram shows.

Code

This snippet mirrors the worked numbers with no dependencies.

1"""Mirror the IQL worked numbers from the page."""2
3import math4
5tau = 0.76beta = 3.07gamma = 0.98
9u_good = 6.0 - 5.010u_bad = 4.0 - 5.011loss_good = (tau if u_good >= 0 else 1 - tau) * u_good**212loss_bad = (tau if u_bad >= 0 else 1 - tau) * u_bad**213print(round(loss_good, 4), round(loss_bad, 4))14# -> 0.7 0.315
16print(round(1.0 + gamma * 4.5, 2))17# -> 5.0518
19print(round(math.exp(beta * 1.0), 4), round(math.exp(beta * -1.0), 4))20# -> 20.0855 0.0498

Watch Out For

Pushing tau to 1.0 on noisy logs

High $\tau$ hugs the best logged return. On noisy data that best return is often a lucky outlier, so $V$ chases noise and targets wobble. The symptom is strong training curves on clean demos but shaky values on mixed logs.

Stay near $\tau = 0.7$ , move to $0.9$ only for expert heavy data, and watch the gap between $V$ and median $Q$ . If $V$ sits far above most logged $Q$ values, lower $\tau$ before touching anything else.

Letting exp beta advantage explode

Large $\beta$ turns weights into spikes. One transition dominates the batch, gradients jump, and the policy collapses onto a few demos. The symptom is NaN loss or a policy that repeats one move.

Keep $\beta$ in $[3.0, 10.0]$ and clip weights at $100.0$ . If the top weight hits the clip often, lower $\beta$ instead of raising the clip.

The Quick Version

IQL trains offline with no queries to unseen actions.
Expectile loss with $\tau = 0.7$ fits $V$ toward good logged $Q$ values, here losses $0.70$ versus $0.30$ .
Q targets use $r + \gamma V(s')$ , here $5.05$ , with no max over actions.
Advantage weights $\exp(\beta A)$ copy winners, here $20.0855$ versus $0.0498$ .
Tune $\tau$ and $\beta$ carefully and clip weights, or noise and overflow undo the stability.