Conditional Value at Risk (CVaR) Optimization

CVaR optimization trains a policy for its bad days, not its average day. It maximizes the average return of the worst alpha fraction of episodes, so rare crashes count more than lucky wins.

CVaR averages the worst alpha tail of the return distribution while VaR only marks where that tail starts

Why Does This Exist?

Standard RL chases the mean. It picks the policy with the highest expected return and stops. That works until the mean hides a cliff.

Picture two delivery-robot policies. Both average +4.3 per shift. One never drops below -2. The other scores +22 on lucky shifts and -20 when it tips down stairs. A mean-only optimizer calls them tied. Your hardware budget does not. You want the policy whose worst fifth of shifts still pays, even if its highlight reel looks plainer.

That is what this page covers. You will see how VaR marks the edge of the bad tail and CVaR averages everything past that edge, with a 10-episode example you can check by hand. Then it sketches how a policy gradient lifts that tail.

Two neighbours do the adjacent jobs. Quantile regression DQN shows how to estimate the return distribution that VaR and CVaR read from. Safe RL shows how to add hard constraints during training and deployment. This page teaches the tail objective itself. For the fixed-grid distribution behind this family see categorical DQN.

You need one setup idea. Each episode produces a total discounted return $R$ . Run one policy ten times and you get ten different $R$ values because starts, noise, and slips differ. VaR and CVaR just summarize the low end of those numbers.

Think of It Like This

The ferry captain judged on stormy crossings

You run a harbor ferry. Each crossing earns a score: punctual arrivals score high, late or rough ones score low.

The owner offers two bonus schemes. Scheme A pays on your average over all 100 crossings. Scheme B pays on your average over your worst 20. Under A you chase calm-day speed records. Under B you practice docking in wind, add fenders, and slow down when chop builds, because one -20 day drags your bonus more than three +22 days lift it.

Map it point for point: crossings are episodes, scores are returns $R$ , the cutoff for your worst 20 is VaR, and the average of those 20 is CVaR. Optimizing CVaR means choosing scheme B.

Where it stops: ferry crossings are close to independent, while RL episodes come from one policy you keep changing. Push the tail up and the whole distribution shifts, so CVaR training needs fresh rollouts instead of one fixed logbook.

How It Actually Works

Let $R$ be the random episodic return under policy $\pi_\theta$ , where $\theta$ holds the policy weights. Let $F(r) = P(R \le r)$ be its cumulative distribution. Let $\alpha$ in $(0, 1)$ be the tail fraction you care about, here $0.2$ for the worst fifth.

VaR marks the edge, CVaR averages past it

VaR (Value at Risk) is the quantile at $\alpha$ :

VaR_\alpha = \inf \{ r : F(r) \ge \alpha \}

In plain words, about 20 percent of episodes score at or below $VaR_{0.2}$ .

CVaR (Conditional Value at Risk) is the mean inside that tail:

CVaR_\alpha = E[R \mid R \le VaR_\alpha] = \frac{1}{\alpha} \int_0^\alpha VaR_u \, du

VaR is one number on the edge; CVaR averages the whole bad slice. For returns, CVaR sits at or below VaR. VaR tells you where the cliff starts, CVaR tells you how deep the drop is.

Fix one convention trap now. Finance often defines CVaR on losses with $\alpha = 0.95$ and averages the worst 5 percent above a high quantile. Here we define it on returns and average the worst $\alpha$ below a low quantile. Same idea, mirrored axis. Always check which side the author averages.

Optimization uses a handy form from Rockafellar and Uryasev. For returns:

CVaR_\alpha = \max_\eta \left\{ \eta - \frac{1}{\alpha} E[(\eta - R)_+] \right\}

Here $\eta$ is a free threshold guess and $(\eta - R)_+ = \max(0, \eta - R)$ is the shortfall below it. The best $\eta$ lands on $VaR_\alpha$ . That form matters because it turns a conditional average into an expectation you can sample.

Worked example with 10 sampled returns

One policy produces these 10 episode returns, in the order they happened:

[12, 5, -8, 22, 3, -20, 9, 15, -2, 7]

Sorted lowest first:

[-20, -8, -2, 3, 5, 7, 9, 12, 15, 22]

The plain mean is $43 / 10 = 4.3$ , since the values sum to $43$ .

Take $\alpha = 0.2$ . The tail holds $0.2 \times 10 = 2$ episodes. By the order-statistic rule VaR is the second smallest value:

VaR_{0.2} = -8

CVaR averages the two tail values:

CVaR_{0.2} = \frac{-20 + (-8)}{2} = \frac{-28}{2} = -14

So the policy averages +4.3 overall but -14 across its worst fifth. That gap is the whole argument: the mean says fine, the tail says fix the stairs fall first. The diagram draws these same numbers, with VaR as the dashed edge and CVaR as the solid tail mean left of it.

How a policy gradient lifts the tail

A CVaR policy gradient keeps the tail and mostly ignores the rest:

Roll out $N$ episodes and collect returns $R_1$ to $R_N$ .
Estimate $VaR_\alpha$ as the empirical $\alpha$ -quantile, exactly as above.
Keep only episodes with $R_i \le VaR_\alpha$ and move $\theta$ to make those episodes likelier and higher scoring, weighted by shortfall times the score gradient $\nabla_\theta \log P_\theta$ .

The indicator $\mathbb{I}(R_i \le VaR_\alpha)$ zeroes out non-tail episodes, which is why only the bad slice teaches. The threshold acts as a built-in baseline, so episodes exactly at VaR contribute nothing.

This costs data. With $\alpha = 0.2$ you discard four of every five episodes per update, so gradients wobble. Teams often warm-start on the mean before tightening $\alpha$ , and read quantiles off a distributional critic such as quantile regression DQN instead of re-sorting thousands of rollouts. Pick CVaR when rare failures dominate your true cost, such as hardware damage. Keep the mean when failures are cheap and every sample must count.

Code

The snippet below computes the same VaR and CVaR from the worked example. Sort, cut the tail, average it.

1from typing import List, Tuple2
3
4def var_cvar(returns: List[float], alpha: float = 0.2) -> Tuple[float, float, float]:5    """Return (mean, VaR_alpha, CVaR_alpha) for lower-tail returns."""6    ordered = sorted(returns)7    n = len(ordered)8    mean = sum(ordered) / n9    k = max(1, int(-(-alpha * n // 1)))  # ceil(alpha * n)10    var = ordered[k - 1]11    tail = ordered[:k]12    cvar = sum(tail) / len(tail)13    return mean, var, cvar14
15
16episode_returns = [12, 5, -8, 22, 3, -20, 9, 15, -2, 7]17mean, var, cvar = var_cvar(episode_returns, alpha=0.2)18print(f"sorted: {sorted(episode_returns)}")19# -> sorted: [-20, -8, -2, 3, 5, 7, 9, 12, 15, 22]20print(f"mean: {mean:.1f}")21# -> mean: 4.322print(f"VaR_0.2: {var}")23# -> VaR_0.2: -824print(f"CVaR_0.2: {cvar:.1f}")25# -> CVaR_0.2: -14.0

Watch Out For

Alpha points opposite ways in finance and RL

Finance papers average losses above a high quantile, so conservative means $\alpha = 0.95$ or $0.99$ . Return-based RL averages gains below a low quantile, so conservative means $\alpha = 0.1$ or $0.2$ . Copy 0.95 into RL code and your tail holds nearly everything, so CVaR collapses to the mean and the extra machinery does nothing.

Symptom: CVaR training behaves exactly like mean training and the tail never improves. Fix: name the side in a comment, such as lower-tail alpha equals 0.2, and assert the kept fraction matches, for example 2 of 10 episodes for alpha 0.2.

CVaR starves on small batches

CVaR learns only from its tail slice. With $\alpha = 0.2$ and 50 episodes per update, just 10 episodes carry gradient and the rest are discarded. Small batches swing wildly: one lucky batch says the tail is fixed, the next says it collapsed.

Symptom: CVaR loss zigzags while the mean looks steady, and evaluation on 20 episodes disagrees with training. Fix: raise episodes per update, warm-start on the mean, then tighten alpha in steps, and read quantiles off a distributional critic when rollouts cost a lot.

The Quick Version

CVaR optimization maximizes the average return of the worst alpha fraction of episodes instead of the overall mean.
VaR at level alpha is the cutoff where that worst slice starts, while CVaR is the average inside the slice.
With returns [12, 5, -8, 22, 3, -20, 9, 15, -2, 7], the mean is 4.3, VaR at 0.2 is minus 8, and CVaR at 0.2 is minus 14.
A CVaR policy gradient keeps only tail episodes and weights each by its shortfall below VaR.
Small alpha values give stronger protection but need larger batches because most episodes are discarded each update.