Evolution Strategies (ES)

Why Does This Exist?

Most policy optimization leans on gradients. You run stochastic gradient descent in RL, which needs a differentiable loss, backpropagation through the policy, and careful handling of moving bootstrap targets. That machinery breaks down when the policy is not differentiable (a simulator with discrete switches, a programmatic controller), when rewards arrive only at the end of a long episode, or when you own thousands of CPUs and a gradient method cannot keep them all fed.

Evolution strategies, in the form Salimans and co-authors described in 2017, take a different deal. Forget derivatives entirely: treat the whole environment as a black box that maps a weight vector to one number, the episode return. Shake the weights with Gaussian noise, run one episode per shaky copy, and move the weights toward the noise that scored well. Each episode is independent, so the work splits across as many parallel workers as you own, and workers swap only random seeds plus scalar returns instead of gradients or weight vectors.

This page covers the Gaussian-perturbation estimator, the two variance tricks that make it practical (antithetic sampling and rank shaping), and why it scales. For the gradient-based counterpart, read policy gradient methods. For how ES keeps searching instead of collapsing early, see exploration versus exploitation.

The one idea you need first: a smoothed objective. Instead of maximizing the raw return $F(\theta)$, ES maximizes its average under noise, $J(\theta) = \mathbb{E}[F(\theta + \sigma \epsilon)]$. That average has a gradient you can sample with zero derivatives, which is the whole trick.

Think of It Like This

How It Actually Works

Write $\theta$ for the policy weights (a flat vector with $d$ entries), $F(\theta)$ for the total episode return from running those weights, $\sigma$ for the noise scale (a small positive number), and $\epsilon$ for one perturbation drawn from a standard Gaussian, $\epsilon \sim \mathcal{N}(0, I)$. The smoothed objective $J(\theta) = \mathbb{E}[F(\theta + \sigma \epsilon)]$ has a gradient with a form you can sample without any derivative:

$g = \frac{1}{n \sigma} \sum_{i=1}^{n} F(\theta + \sigma \epsilon_i) \, \epsilon_i$

Each term pushes $\theta$ along $\epsilon_i$ in proportion to the return that perturbation earned. Then $\theta \leftarrow \theta + \alpha g$, with $\alpha$ the step size. Perturb, score, step: that is the whole optimizer.

Why antithetic sampling halves the variance

Instead of $n$ independent perturbations, draw $m$ of them and evaluate both signs, $\theta + \sigma \epsilon_i$ and $\theta - \sigma \epsilon_i$:

$g = \frac{1}{2 m \sigma} \sum_{i=1}^{m} \big(F(\theta + \sigma \epsilon_i) - F(\theta - \sigma \epsilon_i)\big) \, \epsilon_i$

The difference cancels any baseline shift shared by the pair, so a region where every return is uniformly high or low contributes no noise. You get $2m$ episode scores for $m$ random draws, and each pair votes on its axis with the contrast between its two ends.

Why rank shaping tames wild returns

Raw returns are badly behaved: one lucky episode can outscore everything else combined, and adding a constant to all rewards should change nothing but would rescale the step. So in practice $F$ is replaced by a shaped utility $u$ computed from ranks. Sort the $2m$ scores, map the best to a positive utility and the worst to a negative one (centered so the utilities sum to zero), and use $u$ in place of $F$. The update then depends only on ordering, never on reward scale, offsets, or outliers.

Why it scales to thousands of workers

Episodes never interact, so each worker scores its own perturbations alone. The Salimans design adds a shared noise table: every worker holds the same pre-generated Gaussian table and indexes into it with a seed, so workers broadcast only (seed, scalar return) pairs. Communication per episode stays tiny while the population grows into the thousands. The price is sample hunger: ES typically needs far more episodes than a gradient method to reach the same policy, so you pay it when gradients are unavailable or parallelism is nearly free.

Worked example: one antithetic step

Take a 2D toy with return $F(\theta) = -(\theta_1 - 1)^2 - 2(\theta_2 - 0.5)^2 + 5$, best at $[1.0, 0.5]$ with value $5.0$. Start at $\theta_0 = [0.5, -0.3]$ (return $3.470$), set $\sigma = 0.1$ and step size $\alpha = 0.05$, and fix two draws $\epsilon_1 = [1.0, 0.5]$ and $\epsilon_2 = [-0.4, 0.8]$.

Score the four antithetic copies:

• $\theta_0 + 0.1 \, \epsilon_1 = [0.6, -0.25]$, return $3.715$
• $\theta_0 - 0.1 \, \epsilon_1 = [0.4, -0.35]$, return $3.195$
• $\theta_0 + 0.1 \, \epsilon_2 = [0.46, -0.22]$, return $3.6716$
• $\theta_0 - 0.1 \, \epsilon_2 = [0.54, -0.38]$, return $3.2396$

Pair contrasts: $d_1 = 3.715 - 3.195 = 0.520$ and $d_2 = 3.6716 - 3.2396 = 0.432$. The estimate with $m = 2$ pairs is:

$g = \frac{1}{2 \times 2 \times 0.1} \big(0.520 \, \epsilon_1 + 0.432 \, \epsilon_2\big) = 2.5 \times [0.3472, 0.6056] = [0.868, 1.514]$

Step once: $\theta_1 = [0.5, -0.3] + 0.05 \times [0.868, 1.514] = [0.5434, -0.2243]$. The new return is $3.742$, up from $3.470$. The estimate points the same way as the true gradient $[1.0, 3.2]$ (both entries positive) with sampling noise on the magnitudes: that is the ES bargain in miniature. With only ranks kept, the ordering $F_1^+ > F_2^+ > F_2^- > F_1^-$ alone would drive the same directional vote.

Code

A pure-Python transcription of the worked example. It scores the four antithetic copies, forms the estimate, and takes one step.

1def toy_return(theta):2    """Quadratic toy return. Best at [1.0, 0.5], where it equals 5.0."""3    return -((theta[0] - 1.0) ** 2) - 2.0 * ((theta[1] - 0.5) ** 2) + 5.04
5
6def es_step(theta, perturbations, sigma, step_size):7    """One antithetic ES update: score plus/minus sigma times e, then step."""8    pairs = []9    for e in perturbations:10        plus = toy_return([theta[0] + sigma * e[0], theta[1] + sigma * e[1]])11        minus = toy_return([theta[0] - sigma * e[0], theta[1] - sigma * e[1]])12        pairs.append((plus, minus))13    gradient = [14        sum((plus - minus) * e[i] for (plus, minus), e in zip(pairs, perturbations))15        / (2.0 * len(perturbations) * sigma)16        for i in range(2)17    ]18    updated = [theta[i] + step_size * gradient[i] for i in range(2)]19    return pairs, gradient, updated20
21
22theta0 = [0.5, -0.3]23pairs, grad, theta1 = es_step(theta0, [[1.0, 0.5], [-0.4, 0.8]], sigma=0.1, step_size=0.05)24
25print([round(v, 4) for v in pairs[0]])  # -> [3.715, 3.195]26print([round(v, 4) for v in pairs[1]])  # -> [3.6716, 3.2396]27print([round(v, 3) for v in grad])  # -> [0.868, 1.514]28print([round(v, 4) for v in theta1])  # -> [0.5434, -0.2243]29print(round(toy_return(theta0), 3), round(toy_return(theta1), 3))  # -> 3.47 3.742

Watch Out For

The Quick Version

• Evolution strategies optimize a policy with no derivatives: they score Gaussian perturbations of the weights and step toward the noise that earned the best returns.
• The antithetic estimator scores $\theta + \sigma \epsilon$ and $\theta - \sigma \epsilon$ per draw, so shared baseline shifts cancel and each pair votes with its contrast.
• Rank shaping replaces raw returns with centered rank utilities, making the update immune to reward scale, offsets, and one lucky outlier episode.
• Workers share a noise table and swap only seeds plus scalar returns, which lets ES use thousands of parallel CPUs with tiny communication.
• The trade is sample hunger: ES needs far more episodes than gradient methods, so reach for it when the policy is non-differentiable or parallelism is nearly free.

What to Read Next

• Stochastic Gradient Descent in RL
• Policy Gradient Methods
• Exploration vs. Exploitation
• NEAT: Evolving Neural Topologies