Batch Constrained Q-Learning (BCQ)

Why Does This Exist?

Offline reinforcement learning learns from a fixed log $\mathcal{D} = \{(s, a, r, s')\}$ with no fresh environment interaction. That is the only option when interaction is slow, costly, or unsafe, and it is exactly where standard off policy methods break. The background on the setting is in offline batch reinforcement learning and the failure pattern is named in distributional shift in offline RL.

The break has a name: extrapolation error. A Bellman backup queries $\max_{a'} Q(s', a')$, and in continuous control the actor actively hunts for the action with the largest predicted value. For actions far from the logged data, the critic has seen no ground truth, so its output is an accident of function approximation. The actor latches onto the tallest hallucination, the backup bakes that fantasy into earlier states, and the deployed policy drives off the data manifold. Online learning would correct this by visiting the suspect action and observing the truth. Offline learning cannot.

Batch Constrained Q-Learning (Fujimoto, Meger, and Precup, 2019) was the first deep method built to stop this loop. It never asks the critic about arbitrary actions. It samples candidate actions from a generative model of the logged behavior, allows only tiny bounded adjustments, and scores candidates pessimistically. A later, more explicitly pessimistic alternative is conservative Q-learning.

Think of It Like This

How It Actually Works

BCQ has three coupled parts. $G_\omega$ is a conditional variational autoencoder modeling logged actions $P_{\mathcal{D}}(a \mid s)$. $\xi_\phi(s, a)$ is a perturbation network capped by $\Phi$. $Q_{\theta_1}$ and $Q_{\theta_2}$ are twin critics. Here $s'$ is the next state, $a_i$ are sampled candidates, $\tilde{a}_i$ are perturbed candidates, $r$ is the reward, and $\gamma$ is the discount factor.

Candidate generation from the behavior model

The C-VAE trains on logged pairs $(s, a)$ with an encoder $q(z \mid s, a)$ and decoder $D(s, z)$. At decision time BCQ draws $N$ latent codes $z_i$ from a standard Gaussian and decodes $a_i = D(s', z_i)$. Because the decoder only ever learned logged behavior, these candidates sit on the data manifold by construction. Sampling more candidates (commonly $N = 10$ to $100$) covers multimodal behavior better at the cost of more critic evaluations.

Bounded perturbation

Pure sampling would only clone the logger. BCQ adds $\tilde{a} = a + \xi_\phi(s, a)$ with $\xi_\phi(s, a) = \Phi \tanh(\text{MLP}_\phi(s, a))$, so no tweak exceeds $\pm \Phi$ in each action dimension. A typical setting is $\Phi = 0.05$ times the action range. The perturbation network trains to maximize $Q_{\theta_1}(s, a + \xi_\phi(s, a))$, which is local hill climbing on a short leash: enough to beat a mediocre logger, too short to wander into unsupported regions.

Pessimistic twin critic target

Each perturbed candidate is scored with $\min_j Q_{\theta_j}(s', \tilde{a}_i)$, the lower of the two critic reads. The policy picks the argmax of that minimum, and the Bellman target mixes min and max with weight $\lambda$ in $[0.75, 1.0]$:

$y = r + \gamma \max_i \left[ \lambda \min_j Q'_j(s', \tilde{a}'_i) + (1 - \lambda) \max_j Q'_j(s', \tilde{a}'_i) \right]$

With $\lambda = 1.0$ the target is the pure minimum, the most conservative choice.

Worked example

Logged actions near state $s'$ concentrate in $[0.8, 1.2]$. Fix $r = 1.5$, $\gamma = 0.95$, $\lambda = 1.0$, $\Phi = 0.05$, and $N = 3$.

• The C-VAE samples $a_1 = 0.90$, $a_2 = 1.05$, $a_3 = 1.15$. All three sit inside the logged band.
• Perturbations: $+0.02$, $-0.01$, $+0.03$, all within $\pm 0.05$. Perturbed actions are $\tilde{a}_1 = 0.92$, $\tilde{a}_2 = 1.04$, $\tilde{a}_3 = 1.18$.
• Twin critic reads: $(3.80, 4.00)$ for $0.92$ giving minimum $3.80$; $(4.50, 4.30)$ for $1.04$ giving minimum $4.30$; $(4.10, 4.20)$ for $1.18$ giving minimum $4.10$.
• The policy picks $\tilde{a}_2 = 1.04$ with value $4.30$.
• The target is $y = 1.5 + 0.95 \times 4.30 = 1.5 + 4.085 = 5.585$.

An unconstrained actor would instead query an unseen action such as $a = 2.50$, where one critic hallucinates $8.00$ and the target inflates to $1.5 + 0.95 \times 8.0 = 9.10$. BCQ never samples $2.50$, so the hallucination never enters the update. The diagram draws the three valid candidates, the rejected $2.50$, and the winning target $5.585$.

Code

The snippet replays the same selection and target so you can verify the arithmetic.

1def conservative_value(q1: float, q2: float, lam: float = 1.0) -> float:2    """BCQ critic mix: lam * min + (1 - lam) * max."""3    return lam * min(q1, q2) + (1.0 - lam) * max(q1, q2)4
5
6if __name__ == "__main__":7    raw = [0.90, 1.05, 1.15]8    deltas = [0.02, -0.01, 0.03]9    perturbed = [round(a + d, 4) for a, d in zip(raw, deltas)]10    reads = [(3.80, 4.00), (4.50, 4.30), (4.10, 4.20)]11    scored = [conservative_value(q1, q2) for q1, q2 in reads]12    best = max(range(3), key=lambda i: scored[i])13    target = 1.5 + 0.95 * scored[best]14    print(perturbed)15    print(scored)16    print(perturbed[best], round(scored[best], 2), round(target, 4))

1# -> expected output:2[0.92, 1.04, 1.18]3[3.8, 4.3, 4.1]41.04 4.3 5.585

Watch Out For

The Quick Version

• BCQ is an offline method that restricts every critic query to actions sampled from a generative model of the logged behavior.
• A conditional VAE supplies $N$ in support candidates, so out of distribution hallucinations are never sampled.
• A perturbation network adds tweaks capped at $\pm \Phi$, giving local improvement without leaving the data manifold.
• Twin critics score candidates with the minimum, and the policy takes the argmax of that pessimistic read.
• In the worked example, candidates 0.90, 1.05, and 1.15 become 0.92, 1.04, and 1.18, and the winner 1.04 gives target 5.585.

What to Read Next

• Offline / Batch Reinforcement Learning
• Conservative Q-Learning (CQL)
• Distributional Shift in Offline RL