Risk-Sensitive RL

Risk-sensitive RL picks policies by the shape of their return distribution, not just the average, so a steady policy can beat a flashy gamble with the same mean.

Two policies share mean return 10, but the narrow one scores 9.8 and the gamble scores minus 10.0 under a variance penalty

Why Does This Exist?

Standard RL maximizes the mean. It learns the action with the highest average outcome $Q(s, a) = \mathbb{E}[Z(s, a)]$ , where $Z(s, a)$ is the full distribution over future returns and $Q$ is only its mean. That works when bad outcomes are cheap. It breaks when one rare failure matters more than a slightly better average, such as a trading bot that earns a little extra most days but wipes the account one day in twenty.

The mean erases that difference. Two policies can share exactly the same expected return while one fails badly a quarter of the time. A mean-only agent cannot tell them apart, so it may ship the dangerous one with confidence.

This page covers objectives that score the shape of the distribution: variance penalties, quantile targets, and CVaR. It does not re-teach how those distributions get learned. Quantile learning lives in quantile regression DQN, categorical distributions live in categorical DQN, and hard constraints live in safe RL.

The one idea you need first is a return distribution. Write the return of a policy as a random variable $Z$ . Its mean $\mathbb{E}[Z]$ is the usual target. Its variance $\mathrm{Var}(Z)$ measures spread around the mean. Its lower quantile $q_{0.1}$ is the value beaten 90 percent of the time. Its CVaR at level 0.25 is the average over the worst quarter of outcomes. Risk-sensitive objectives replace the mean with one of these tail aware scores.

Think of It Like This

Two commutes with the same average

Route A takes 30 minutes every day. Route B takes 15 minutes nine days in ten and 165 minutes on the tenth day when the bridge jams. Check the average: $0.9 \times 15 + 0.1 \times 165 = 13.5 + 16.5 = 30$ . Same average, very different life.

On interview day you take route A without thinking. You care about the worst case, not the average. On a lazy Saturday you might take route B for the fast run. Same roads, opposite choice, because your objective changed. Risk-averse means lifting the floor. Risk-seeking means chasing the ceiling.

Where it stops: you see commute times directly, but an agent must estimate tails from samples. Rare outcomes are rare by definition, so tail estimates wobble. Keep that in mind when you read the second pitfall below.

How It Actually Works

Risk-neutral RL maximizes $J(\pi) = \mathbb{E}[Z^\pi]$ for a policy $\pi$ . Risk-sensitive RL swaps the mean for a functional that reacts to spread and tails. Three workhorses cover most practice.

The three objectives

First, the variance penalty. With $\lambda$ (lambda) the price per unit of variance:

$J(\pi) = \mathbb{E}[Z^\pi] - \lambda \mathrm{Var}(Z^\pi)$

Here $\lambda$ above 0 punishes spread. It is simple and smooth, but it punishes upside surprises as much as disasters.

Second, the quantile objective. Pick $\tau$ (tau) between 0 and 1 and maximize the $\tau$ quantile $q_\tau$ , the return beaten with probability $1 - \tau$ . Low quantiles such as $q_{0.1}$ are pessimistic and risk-averse. High quantiles such as $q_{0.9}$ are optimistic and risk-seeking, right for settings where only the best run counts.

Third, CVaR. Write $\text{CVaR}_{0.25}$ for the mean over the worst quarter of outcomes. Maximizing it means maximizing the floor rather than the average. If you already have quantiles from quantile regression DQN, CVaR is the average of the bottom ones. Risk sensitivity is a choice of what to optimize, layered on the distributional view.

Worked example

Policy A is steady with returns 8 at probability 0.25, 10 at 0.5, and 12 at 0.25. Policy B gambles with minus 10 at 0.25, 10 at 0.5, and 30 at 0.25.

Means are both 10:

$\mathbb{E}[Z_A] = 0.25 \times 8 + 0.5 \times 10 + 0.25 \times 12 = 2 + 5 + 3 = 10$

$\mathbb{E}[Z_B] = 0.25 \times (-10) + 0.5 \times 10 + 0.25 \times 30 = -2.5 + 5 + 7.5 = 10$

Second moments are $\mathbb{E}[Z_A^2] = 16 + 50 + 36 = 102$ and $\mathbb{E}[Z_B^2] = 25 + 50 + 225 = 300$ . Variances are:

$\mathrm{Var}(Z_A) = 102 - 100 = 2, \qquad \mathrm{Var}(Z_B) = 300 - 100 = 200$

The worst quarter is exactly the bottom atom in each case, so $\text{CVaR}_{0.25}(A) = 8$ and $\text{CVaR}_{0.25}(B) = -10$ . With $\lambda = 0.1$ :

$J(A) = 10 - 0.1 \times 2 = 9.8, \qquad J(B) = 10 - 0.1 \times 200 = -10.0$

Mean-only ties them at 10 to 10. Variance penalty and CVaR both pick A by a landslide. A risk-seeking $q_{0.9}$ objective picks B, since B returns 30 in its top quarter against 12 for A. The diagram draws these two distributions on one axis with the same mean line and different tails.

Code

Scores both policies on mean, variance, CVaR at 25 percent, and the penalized objective. Same numbers as the hand trace:

1from typing import Dict, List, Tuple2
3
4def cvar(outcomes: List[Tuple[float, float]], alpha: float = 0.25) -> float:5    """Mean over the worst alpha fraction. Outcomes are (value, prob) pairs."""6    need = alpha7    total = 0.08    for x, p in sorted(outcomes):9        take = min(p, need)10        total += take * x11        need -= take12        if need <= 0:13            break14    return total / alpha15
16
17def score(outcomes: List[Tuple[float, float]], lam: float = 0.1) -> Dict[str, float]:18    mean = sum(x * p for x, p in outcomes)19    var = sum(p * (x - mean) ** 2 for x, p in outcomes)20    return {"mean": mean, "var": var, "cvar25": cvar(outcomes), "penalized": mean - lam * var}21
22
23if __name__ == "__main__":24    steady = [(8.0, 0.25), (10.0, 0.5), (12.0, 0.25)]25    gamble = [(-10.0, 0.25), (10.0, 0.5), (30.0, 0.25)]26    print(score(steady))27    # -> {'mean': 10.0, 'var': 2.0, 'cvar25': 8.0, 'penalized': 9.8}28    print(score(gamble))29    # -> {'mean': 10.0, 'var': 200.0, 'cvar25': -10.0, 'penalized': -10.0}

Both prints match the hand arithmetic: shared mean 10, variances 2 versus 200, CVaR 8 versus minus 10, penalized 9.8 versus minus 10.0.

Watch Out For

Evaluating on the mean and shipping the gamble

You train two checkpoints, each averages about 10 over 50 eval episodes, and you ship the slightly higher one. In production its tail contains account wiping episodes the incumbent never had. The dashboard looked identical because the dashboard only logged means.

Fix it by reporting tails beside means on the same eval episodes: variance and $\text{CVaR}_{0.25}$ from the snippet above. Select checkpoints on the risk objective you actually care about, not on the mean you happened to log.

Trusting a tail built on a handful of episodes

$\text{CVaR}_{0.05}$ from 100 episodes rests on 5 episodes. One lucky run flips which policy looks safer, and the ranking reverses on the next eval. The symptom is a risk ranking that will not replicate across days with identical code.

Fix it with more episodes, wider error bars, or a coarser level. Prefer $\text{CVaR}_{0.25}$ over $\text{CVaR}_{0.05}$ on small eval budgets and always attach the sample count to a tail number. A tail without a sample count is a rumor.

The Quick Version

Mean-only RL cannot separate a steady policy from a gamble with the same average.
Risk-sensitive RL scores spread and tails with variance penalties, quantiles, or CVaR.
Both example policies average 10, but variances are 2 versus 200 and CVaR at 25 percent is 8 versus minus 10.
A variance penalty at lambda 0.1 scores them 9.8 versus minus 10.0, turning a tie into a clear pick.
Risk-averse lifts the floor with low quantiles and CVaR. Risk-seeking chases the ceiling with high quantiles.
Report tails next to means, and never trust a tail estimated from a handful of episodes.