Offline Reinforcement Learning

Offline RL trains from logged transitions with no live steps, by pushing down unseen action values and by generating actions from high return examples.

CQL scores a penalty of 3.143 from a soft maximum of 6.143 minus logged value 3.00, pushing the unseen action down and holding the target at 5.50

Why Does This Exist?

Live trials fail in clinics, cars, plants, and large recommender fleets. Logs remain. Offline RL turns those logs into a policy with no environment calls during training. Running DQN or SAC as is on logs fails because the Bellman max queries moves the logs never show, and nets extrapolate with false highs.

Two fixes now carry most work. Conservative Q-Learning adds a penalty that lower bounds value by pushing down all actions and pushing up logged ones. Decision Transformer drops Bellman math and copies high return sequences conditioned on a target return. Start with conservative Q-learning for the penalty view, implicit Q-learning for the inside support view, and decision transformers for the sequence view.

This page covers both fixes with one traced CQL penalty.

Think of It Like This

Studying chess only from past tournament books

A student reads 10,000 grandmaster games with no board and no coach.

A naive reader sees an unplayed rim knight move and thinks it must be a secret win because no book refutes it. A careful reader treats unplayed moves as suspect and discounts them. A second careful reader ignores novelties and replays only high scoring lines move by move when asked for a win.

The map holds. Books are logs. Suspect novelties are out of distribution actions. Discounting them is CQL. Replaying high scoring lines for a target score is Decision Transformer.

Where it stops: books carry human sense about blunders, while Q nets need an explicit penalty to distrust gaps. Sequence models also need a reachable target return, not a fantasy score.

How It Actually Works

Logs hold $(s, a, r, s^\prime)$ under past policy $\hat{\pi}_\beta$ . The CQL discrete penalty on state $s$ is:

\mathcal{R} = \log \sum_a \exp Q(s, a) - \mathbb{E}_{a \sim \hat{\pi}_\beta}[Q(s, a)]

Here the first term is a soft maximum over all actions that pushes every $Q$ down, hardest on the current peak. The second term is mean $Q$ under logged actions that pushes seen $Q$ up. Weight $\alpha$ sets strength. For large enough $\alpha$ , learned value lower bounds truth, so policy cannot chase ghosts.

Worked example with real numbers

State $s$ has $Q = [4.0, 2.0, 6.0]$ for $[a_1, a_2, a_3]$ . Logs show $a_1$ and $a_2$ at 0.5 each and $a_3$ at 0.0. So $a_3$ is unseen with a hallucinated 6.0.

Soft maximum: $\exp$ values are about 54.598, 7.389, and 403.429. Sum is 465.416. Log gives:

\mathrm{LSE} = \ln(465.416) \approx 6.143

Logged mean is $0.5 \times 4.0 + 0.5 \times 2.0 = 3.0$ . Penalty is:

\mathcal{R} = 6.143 - 3.0 = 3.143

Softmax shares are $54.598/465.416 \approx 0.117$ , $7.389/465.416 \approx 0.016$ , and $403.429/465.416 \approx 0.867$ . Gradient of the penalty on $Q$ is softmax minus log share, which gives about $[-0.383, -0.484, +0.867]$ . Positive 0.867 on unseen $a_3$ pushes it down during minimization. Negative parts push seen actions up.

With Bellman reward $r = 1.0$ and $\gamma = 0.9$ , the grounded target on seen $a_1$ stays $1.0 + 0.9 \times 5.0 = 5.50$ in the companion trap example, while the unpenalized max would chase the spike.

Sequence view without values

Decision Transformer writes a path as return to go plus states and actions:

\tau = (\hat{R}_1, s_1, a_1, \dots, \hat{R}_T, s_T, a_T)

Here $\hat{R}_t$ sums future rewards from $t$ . Training predicts $a_t$ from past tokens by supervised loss. At test time, set a high but reachable $\hat{R}_1$ , act, observe $r_t$ , then set $\hat{R}_{t+1} = \hat{R}_t - r_t$ . No max over unseen actions appears, so the spike trap never forms.

Code

This snippet repeats the penalty, softmax, and gradients from the worked example.

1import math2
3
4def cql_terms(q: list[float], data: list[float]) -> tuple[float, list[float]]:5    m: float = max(q)6    s: float = sum(math.exp(v - m) for v in q)7    lse: float = m + math.log(s)8    pen: float = lse - sum(d * v for d, v in zip(data, q))9    grad: list[float] = [math.exp(v - m) / s - d for v, d in zip(q, data)]10    return pen, grad11
12
13pen, grad = cql_terms([4.0, 2.0, 6.0], [0.5, 0.5, 0.0])14print(f"{pen:.4f}")  # -> 3.142915print([f"{g:.3f}" for g in grad])  # -> ['-0.383', '-0.484', '0.867']

Watch Out For

Training curves that climb while the robot falls

Unpenalized offline runs report rising Q with no live check. Symptom: logged curves look heroic, deployment fails at once because highs were extrapolation artifacts.

Never trust offline Q curves alone. Track the share of chosen actions that appear in logs, keep a held out log return, and require the penalty to keep unseen peaks below seen picks before deployment.

Asking the transformer for an impossible return

Conditioning on a return above all logs asks the model to invent moves it never saw. Symptom: neat training loss, wandering or frozen rollouts because the prompt sits outside the data.

Set target return near the best logged return for the same start, not the global max. When rollouts undershoot, lower the target until behavior matches logged high performers, then raise it in small steps.

The Quick Version

Offline RL learns only from logged $(s, a, r, s^\prime)$ with no live steps, so unseen actions need explicit distrust.
CQL penalty equals soft maximum minus logged mean $Q$ , here $6.143 - 3.0 = 3.143$ .
Softmax shares of $[4.0, 2.0, 6.0]$ are about $[0.117, 0.016, 0.867]$ , so gradients are $[-0.383, -0.484, +0.867]$ .
Positive gradient on the unseen peak pushes it down, negatives lift logged actions.
Decision Transformer avoids the max by generating actions conditioned on return to go $\hat{R}_t$ .
Ask for reachable returns and check that chosen actions stay inside log support.