Offline / Batch Reinforcement Learning

Offline RL learns a policy from fixed logs with no live steps, by treating unseen actions as guilty until proven safe and stitching good segments together.

Standard Q-learning targets 9.28 by picking an unseen spike of 9.20 while a conservative penalty of 5.00 restores the grounded target of 5.50

Why Does This Exist?

Online RL learns by trying actions and watching what happens. That loop fails where trying is unsafe or dear. A clinic cannot test random doses on patients. A car cannot test a divider crash. A plant cannot test outside safe margins. A trading desk cannot test a flawed order at size.

Those places still hold logs from humans, rules, and older controllers. Write the set as $\mathcal{D} = \{(s_i, a_i, r_i, s^\prime_i)\}$ drawn under a past behavior policy $\pi_\beta$ . Offline RL, also called batch RL, trains only from $\mathcal{D}$ with zero live steps. Standard off policy methods still collapse there, which is why the field exists. The shift problem has its own page at distributional shift in offline RL. The constraint fix lives at batch constrained Q-learning. The methods sequel is offline RL, and the sequence view is decision transformers.

This page covers the trap and the four fix families, with one traced Bellman update.

Think of It Like This

A resident who only reads surgery logs

A resident must learn heart surgery from 50,000 past logs with no live practice.

Copying the average move is cloning. It repeats past slips and freezes on rare cases. Trial and error is banned for clear cause. The offline path is stitching. Doctor A closes well but sutures poorly. Doctor B opens poorly but sutures masterfully. The resident joins the best opening to the best suturing and beats each source alone.

The map holds. Logs are $\mathcal{D}$ . Average copying is behavior cloning. Joining best segments is trajectory stitching through Bellman backups.

Where it stops: humans know some moves are fatal without seeing them fail, while nets can score an unseen cut highly by accident. That is why offline methods penalize unseen actions instead of trusting them.

How It Actually Works

The Bellman target for $Q(s, a)$ uses a max over next actions:

y = r(s, a) + \gamma \max_{a^\prime} Q(s^\prime, a^\prime)

Here $r$ is reward, $\gamma$ is discount, $s^\prime$ is next state, and $a^\prime$ ranges over next actions. Online, a false high $Q$ gets tried, fails, and gets corrected. Offline, no trial can refute it.

The three step trap

First the target queries an action with $\pi_\beta(a^\prime \mid s^\prime)$ near zero. Then the max picks the largest positive noise spike among unseen scores. Then backups spread that hallucinated value to earlier states. Values climb on paper while the deployed policy fails.

Four fix families

Policy constraint keeps $\pi(a \mid s)$ near behavior $\pi_\beta(a \mid s)$ within a divergence budget, as in BCQ style methods. Conservative value keeps the policy free but pushes down unseen $Q$ and pushes up seen $Q$ , as in CQL style losses. In sample learning never queries unseen actions and fits an upper expectile inside support, as in IQL with expectile around 0.7 to 0.9. Sequence modeling drops Bellman updates and predicts actions conditioned on return to go, as in Decision Transformer style models.

Worked example with real numbers

Take transition with $r = 1.0$ and $\gamma = 0.9$ . Next state has three scores. Seen action scores $5.0$ . Unseen actions score $9.2$ and $2.1$ . Truth on seen data is $5.0$ .

Standard max picks $9.2$ :

y_{std} = 1.0 + 0.9 \times 9.2 = 1.0 + 8.28 = 9.28

Grounded target is $1.0 + 0.9 \times 5.0 = 5.50$ . Overestimate is $9.28 - 5.50 = 3.78$ .

A conservative penalty $\alpha = 5.0$ subtracts 5.0 from unseen scores only. Adjusted scores are $5.0$ seen, $9.2 - 5.0 = 4.2$ , and $2.1 - 5.0 = -2.9$ . The max now picks the seen action:

y_{cql} = 1.0 + 0.9 \times 5.0 = 5.50

Error falls to $5.50 - 5.50 = 0.00$ . The spike is still stored, but it no longer wins.

Code

This snippet repeats the standard and conservative targets from the worked example.

1def bellman_target(reward: float, gamma: float, best_q: float) -> float:2    return reward + gamma * best_q3
4
5std: float = bellman_target(1.0, 0.9, 9.2)6cql: float = bellman_target(1.0, 0.9, 5.0)7print(f"{std:.2f}")  # -> 9.288print(f"{cql:.2f}")  # -> 5.509print(f"{std - 5.5:.2f}")  # -> 3.7810print(f"{cql - 5.5:.2f}")  # -> 0.00

Watch Out For

Mistaking offline RL for cloning

Assuming no exploration means no improvement leads teams to clone logs. Symptom: the policy copies mixed skill demonstrators, repeats their slips, and never beats the best log.

Look for stitching. When one path runs A to B well and another runs B to C well, Bellman backups can join them through shared $V(B)$ . Clone as a baseline, then run a conservative or in sample method and compare against best trajectory return, not mean log return.

Over conservatism that freezes the policy

A very large penalty or tight constraint removes the trap but also removes gain. Symptom: values look safe, yet the policy matches the behavior clone and ignores clearly good moves inside support.

Tune the penalty until held out seen returns rise without unseen scores winning. When raising $\alpha$ stops changing chosen actions but keeps lowering values, you have passed the useful range.

The Quick Version

Offline RL trains only from fixed logs $\mathcal{D}$ under past policy $\pi_\beta$ , with no live environment steps.
Standard Bellman max queries unseen actions, picks noise spikes, and spreads false value backward.
The worked update targets 9.28 on a 9.20 spike versus a grounded 5.50, an overestimate of 3.78.
A penalty of 5.00 on unseen scores restores the seen pick and a target of 5.50 with zero error.
Four fixes cover the space: policy constraint, conservative values, in sample learning, and return conditioned sequences.
Unlike cloning, offline RL can stitch best segments from mixed logs into a policy that beats each source.