Distributional Shift in Offline RL

You train from a fixed log with no chance to try a new move, so one flattering guess about an action you never saw can poison every value behind it.

One unseen action scored at 3.5 against a seen 2.0 hijacks the Bellman max, lifts its predecessor target to 4.15, and splits estimated value from true return

Why Does This Exist?

Some places cannot afford trial and error. A hospital, a trading desk, a fleet of trucks. You have logs from past behavior and you must learn a better policy without running a new experiment. That is offline RL: train from a fixed dataset of tuples $(s, a, r, s^{\prime})$ , where $s$ is the state, $a$ the action taken, $r$ the reward seen, and $s^{\prime}$ the next state.

Standard off policy methods break here. Deep Q-networks and actor critic methods assume they can test a guess and get corrected. Offline you cannot. So small estimation errors on actions you never tried do not get refuted. They get amplified.

This page names that failure: distributional shift, the gap between the behavior policy $\pi_{\beta}$ that logged the data and the learned policy $\pi$ you are optimizing. Fixes live next door. Batch-constrained Q-learning stays near logged actions, conservative Q-learning pushes unseen values down, and implicit Q-learning avoids asking about unseen actions at all. Start with offline batch RL for the setup, then read those fixes.

Think of It Like This

A kitchen that only serves what it has tasted

You inherit a restaurant kitchen with a thick book of past orders and reviews. Every dish in the book has a reliable rating. Your job is to write a better menu without cooking anything new.

You notice a blank page at the back. Someone scribbled a dish nobody ever cooked, with a perfect score in the margin. A greedy rule says to build the whole menu around it. You print it, customers order it, and the kitchen cannot make it. The dining room empties.

That scribbled score is an out of distribution Q-value. Online training would cook the dish once, taste the failure, and cross it out. Offline training has no stove, so the fantasy score survives and drags every course around it down. The analogy stops where cooking starts: a chef knows an untasted dish is suspect, while a network treats its own extrapolation as fact.

How It Actually Works

Call the logged state action density $d^{\pi_{\beta}}(s, a)$ and the density your new policy would visit $d^{\pi}(s, a)$ . The moment $\pi$ differs from $\pi_{\beta}$ , you query the Q-function where $d^{\pi_{\beta}}(s, a)$ is near zero. Networks interpolate well and extrapolate wildly, so $Q(s, a)$ on those queries carries large error.

The Bellman target makes it worse because it takes a maximum over next actions:

y(s, a) = r(s, a) + \gamma \max_{a^{\prime}} Q(s^{\prime}, a^{\prime})

Here $\gamma$ is the discount factor between 0 and 1. The max scans every candidate action, including ones absent from the data, and keeps the largest value. If even one unseen action has a lucky positive error, the max keeps exactly that one. An actor trained to maximize $Q$ then hunts for such peaks on purpose.

Online, the cure is automatic. You take the suspect action, observe a poor reward, and the temporal difference error drags its value down. Offline, that correction never arrives. The inflated target trains its predecessors, which inflate theirs, so error flows backward through the trajectory while true performance falls. That is the same unstable trio described in the deadly triad: function approximation plus bootstrapping plus off policy data with no correction.

Worked example

Take a two step chain with discount $\gamma = 0.9$ . From $s_0$ , action $a_0$ pays $r = 1.0$ and leads to $s_1$ . At $s_1$ there are two actions. The logged action has true value $Q^{\star}(s_1, a_{\text{seen}}) = 2.00$ . The unseen action is poor in reality, $Q^{\star}(s_1, a_{\text{ood}}) = 0.50$ . The network fits the seen action correctly at $2.00$ but noise puts the unseen one at $3.50$ .

The Bellman backup at $s_1$ takes the max:

\max_{a^{\prime}} Q(s_1, a^{\prime}) = \max(2.00, 3.50) = 3.50

So the target for the predecessor becomes:

y(s_0, a_0) = 1.00 + 0.90 \times 3.50 = 1.00 + 3.15 = 4.15

The honest target uses the logged action only:

y^{\star}(s_0, a_0) = 1.00 + 0.90 \times 2.00 = 1.00 + 1.80 = 2.80

Overestimation at $s_0$ is $4.15 - 2.80 = 1.35$ . The greedy policy picks $a_{\text{ood}}$ at $s_1$ because $3.50$ beats $2.00$ . Deployed, it earns $1.00 + 0.90 \times 0.50 = 1.45$ from $s_0$ while believing $4.15$ . One unseen action, one latch, and belief splits from reality by nearly a factor of three.

Code

This snippet replays the worked example. It computes the unconstrained target, the honest target, and the deployed return from the same four numbers.

1def bellman_target(reward: float, gamma: float, next_q: float) -> float:2    return reward + gamma * next_q3
4
5def main() -> None:6    r, gamma = 1.0, 0.97    q_seen, q_ood = 2.00, 3.508    q_star_seen, q_star_ood = 2.00, 0.509
10    picked = max(q_seen, q_ood)11    target = bellman_target(r, gamma, picked)12    honest = bellman_target(r, gamma, q_star_seen)13    deployed = bellman_target(r, gamma, q_star_ood)14
15    print(f"picked next value: {picked:.2f}")  # -> picked next value: 3.5016    print(f"learned target: {target:.2f}")  # -> learned target: 4.1517    print(f"honest target: {honest:.2f}")  # -> honest target: 2.8018    print(f"deployed return: {deployed:.2f}")  # -> deployed return: 1.4519    print(f"overestimation: {target - honest:.2f}")  # -> overestimation: 1.3520
21
22if __name__ == "__main__":23    main()

Watch Out For

Training longer to fix a collapsing offline policy

More epochs help in supervised learning, so teams run more epochs when offline returns look bad. Here that backfires. Extra passes fit the logged tuples a little better while the unseen peaks get sharper and the Bellman backup recirculates them more times.

You will see training loss fall while deployed return falls with it. Stop trusting the Bellman loss as a progress meter. Hold out trajectories for fitted Q evaluation, cap the policy near logged actions, or penalize unseen values, and judge the run by evaluation estimates instead of loss curves.

The Quick Version

Distributional shift is the gap between the logging policy $\pi_{\beta}$ and the learned policy $\pi$ , measured where data density is near zero.
The Bellman max keeps the largest next action value, so it keeps lucky errors on unseen actions first.
With no environment step to refute them, inflated targets train predecessors and error flows backward.
In the worked chain, a $3.50$ guess on one unseen action lifts the predecessor target to $4.15$ against an honest $2.80$ , while deployment earns $1.45$ .
Longer unconstrained training widens the gap. Constrain the policy, penalize unseen values, or stay in sample.