Conservative Q-Learning (CQL)

CQL trains offline values that refuse to brag about actions the dataset never tried. It pushes unseen actions down and dataset actions up, so the learned values sit safely below the truth.

CQL pulls every action down in proportion to its softmax share while lifting dataset actions, so the 5.0 mirage at a2 falls below the supported 3.0 at a1

Why Does This Exist?

Offline RL learns from a fixed log $\mathcal{D} = \{(s, a, r, s')\}$ with no fresh tries allowed. Standard off-policy updates still take a maximum over next actions:

y = r + \gamma \max_{a'} Q(s', a')

That maximum is the trap. A network with random initialization assigns some value to actions nobody ever took, and one of those guesses is always high. The max picks the hallucination, the next update treats it as ground truth, and values snowball until the deployed policy chases actions that never worked.

Behaviour-policed methods fight this by modelling the logging policy and staying near it, but density models in high dimensions add their own errors. CQL (Kumar et al., 2020) acts on values directly: add a regularizer that depresses unseen actions and props up logged ones, hard enough to guarantee a lower bound on the true values.

This page covers that regularizer and its gradient dynamics. For the shift problem itself see distributional shift in offline RL. For the batch setup see offline batch RL. For the policy-constraint alternative see batch-constrained Q-learning.

You need one setup idea: an offline dataset holds only the actions some past behaviour tried, and anything else is out of distribution with no measured return.

Think of It Like This

A cautious engineer rating a bridge from old logs

Two engineers must certify an old bridge using only a ledger of past crossings.

The first fits a curve through the logged vans and buses, then reads off the untested 60-ton truck row: the curve says it is the safest load yet. The first real truck collapses the bridge.

The second engineer rates every logged weight from measurements and stamps every untested weight far below the weakest logged one. The certificate is pessimistic on purpose, and no truck gets approved on a guess.

Map it point for point: the ledger is the dataset, vehicle weights are actions, the curve is the unconstrained Q-network, and the pessimistic stamp is the CQL penalty pushing unseen actions down.

Where it stops: bridges are static, while RL stitches sequences, so CQL must preserve Bellman backups along logged transitions instead of stamping everything down equally.

How It Actually Works

CQL adds a regularizer $\mathcal{R}(Q)$ with weight $\alpha$ to the usual Bellman squared error:

\min_Q \; \alpha \mathcal{R}(Q) + \frac{1}{2} E_{\mathcal{D}}\left[ (Q - \hat{\mathcal{B}}^\pi \hat{Q})^2 \right]

The practical form uses a log-sum-exp over actions minus the dataset mean:

\mathcal{R}(Q) = E_{s \sim \mathcal{D}}\left[ \log \sum_a \exp Q(s, a) - E_{a \sim \hat{\pi}_\beta}[Q(s, a)] \right]

In continuous spaces the sum is sampled, half uniformly and half from the current policy, with importance weights.

Why the gradient targets spikes

Differentiate with respect to one $Q(s, a)$ :

\nabla_{Q(s,a)} \mathcal{R} = \mu(a \mid s) - \hat{\pi}_\beta(a \mid s), \quad \mu(a \mid s) = \frac{\exp Q(s, a)}{\sum_{a'} \exp Q(s, a')}

Every action feels downward pressure through its softmax share $\mu$ , and only logged actions get the upward refund $\hat{\pi}_\beta$ . The tallest spike owns most of $\mu$ , so it attracts most of the punishment. Logged actions roughly cancel and stay anchored to Bellman targets. Unseen actions get pure downward push.

With $\alpha$ large enough, Kumar et al. prove the learned values lower-bound the truth pointwise, $\hat{V}_{CQL}(s) \le V(s)$ . The policy cannot profit from imagination.

Worked example with 3 actions

State $s$ has actions $\{a_1, a_2, a_3\}$ . Only $a_1$ appears in the dataset, so $\hat{\pi}_\beta = [1.0, 0.0, 0.0]$ . Current values are $Q = [3.00, 5.00, 1.00]$ , with a dangerous mirage at $a_2$ . Bellman target for $a_1$ is $y = 3.00$ . Set $\alpha = 1.00$ and step size $\eta = 0.50$ .

Exponentials: $e^{3} \approx 20.0855$ , $e^{5} \approx 148.4132$ , $e^{1} \approx 2.7183$ , sum $Z \approx 171.2170$ , so $\log Z \approx 5.1429$ . The regularizer is $5.1429 - 3.00 = 2.1429$ .

Softmax shares: $\mu = [0.1173, 0.8668, 0.0159]$ . The mirage owns 87 percent of the penalty mass. Gradients $\mu - \hat{\pi}_\beta$ are $[-0.8827, +0.8668, +0.0159]$ . Negative on $a_1$ means push up; positive elsewhere means push down. The Bellman gradient on $a_1$ is $3.00 - 3.00 = 0.00$ .

One gradient step:

$a_1$ : $3.00 - 0.50 \times (-0.8827) = 3.4414$
$a_2$ : $5.00 - 0.50 \times 0.8668 = 4.5666$
$a_3$ : $1.00 - 0.50 \times 0.0159 = 0.9920$

After five steps values reach $[3.56, 3.37, 0.94]$ . The mirage drops below the supported action, so the greedy pick flips to $a_1$ . The diagram draws these before and after bars.

Code

This snippet computes the same softmax shares, gradients, and five update steps with only the standard library.

1import math2from typing import List3
4
5def softmax(q: List[float]) -> List[float]:6    """Numerically stable softmax over a list of Q-values."""7    top = max(q)8    exps = [math.exp(v - top) for v in q]9    total = sum(exps)10    return [v / total for v in exps]11
12
13q = [3.0, 5.0, 1.0]14mu = softmax(q)15print([round(v, 4) for v in mu])16# -> [0.1173, 0.8668, 0.0159]17grad = [mu[0] - 1.0, mu[1], mu[2]]18print([round(v, 4) for v in grad])19# -> [-0.8827, 0.8668, 0.0159]20
21qq = list(q)22for _ in range(5):23    m = softmax(qq)24    g = [m[0] - 1.0 + (qq[0] - 3.0), m[1], m[2]]25    qq = [qq[i] - 0.5 * g[i] for i in range(3)]26print([round(v, 2) for v in qq])27# -> [3.56, 3.37, 0.94]28print(qq[0] > qq[1])29# -> True

Watch Out For

Alpha so high the agent just copies the log

Crank $\alpha$ to 50 and the penalty buries the Bellman error. Every deviation from logged actions looks terrible, so the policy hugs demonstrations including their mistakes and detours. You lose stitching, which is the whole point of offline RL: combining good segments from different trips into one better route.

Symptom: training looks safe but evaluation never beats the behaviour log. Fix: keep $\alpha$ in $[1.0, 5.0]$ on standard benchmarks, or use a Lagrange variant that auto-tunes $\alpha$ against a fixed penalty budget.

Alpha so low the mirage survives

Set $\alpha$ near zero and the update is plain DQN on a fixed log. The tallest unseen spike keeps winning the max, values inflate, and deployment rediscovers the crash the penalty was meant to stop.

Symptom: offline Q-values climb far past any logged return while logged-action errors stay small. Fix: watch the gap between log-sum-exp and dataset mean during training. If it grows while returns do not, raise $\alpha$ before trusting the policy.

The Quick Version

CQL adds a log-sum-exp penalty over actions minus the dataset action mean to the Bellman loss.
Its gradient pushes each action down by its softmax share and refunds only logged actions, so spikes attract the most punishment.
With values $[3.0, 5.0, 1.0]$ , shares are $[0.1173, 0.8668, 0.0159]$ and one step moves values to $[3.44, 4.57, 0.99]$ .
After five steps values reach $[3.56, 3.37, 0.94]$ , so the greedy pick returns to the supported action.
Large enough alpha guarantees a pointwise lower bound on true values; tune it in $[1.0, 5.0]$ to keep trajectory stitching alive.