Evolutionary Methods and Neuroevolution

Instead of computing gradients, keep a population of policies, score each one, and breed the winners with small random tweaks until behavior improves.

Four policies are scored, the top two are kept as parents, and averaging plus Gaussian mutation builds the next generation without any gradient

Why Does This Exist?

Gradients need a slope to follow, and in reinforcement learning that slope often does not exist. The reward can stay flat for hundreds of moves and then pay out once, so every small nudge of the weights returns zero signal. It can be deceptive, where the nearby improvement points away from the real goal. Or the policy itself can be something you cannot differentiate at all: a discrete rule set, a compiled simulator, a physical robot.

Policy gradient methods estimate a slope from episode returns, and deep reinforcement learning adds neural networks that learn features from raw input. Both still assume that nudging parameters toward higher return gives a usable signal. When the reward stays flat for long stretches, that nudge carries no information and learning stalls. That flat stretch is the same problem behind exploration versus exploitation: you must try far-apart behaviors before you can refine any of them.

Evolution takes a different deal. Skip derivatives entirely. Keep a population of full policies, run each one, score it by episode return, keep the winners, mix them, add random tweaks, and repeat. It works with any reward function and any policy form because it only looks at scores, never at slopes.

This page teaches the shared loop: selection, crossover, and mutation. It also sketches three specialized children so you can tell them apart: Evolution Strategies, which samples Gaussian noise around a mean policy; CMA-ES, which adapts the shape of that sampling distribution as it learns; and NEAT, which grows network weights and network topology together starting from small nets.

The one idea you need first: fitness means episode return here. One number per policy, higher is better: total reward from a full run. Everything else is machinery for producing policies with bigger numbers.

Think of It Like This

Saving seeds from your best tomato plants

You plant four tomato rows in spring. At harvest you taste each row and weigh the yield. You save seeds only from the two tastiest, most productive rows. Next spring you plant those saved seeds, cross two of the best plants to mix traits, and accept that a little natural variation shows up on its own. You also replant your single best row unchanged, in case the new mixes flop.

The map is direct. The four rows are the population. Taste plus yield is fitness, the episode return. Saving two rows is truncation selection. Mixing two plants is crossover. Natural variation is mutation. Replanting the best row unchanged is elitism.

Where it stops: your garden gives one harvest a year with weather noise you cannot control. Here you choose the fitness function and the tweak size, usually a Gaussian spread you shrink over time, and you test every candidate from the same start, which no garden lets you do.

How It Actually Works

Call the policy weights $\theta$ : one flat vector holding every number the policy needs, whether that is four scalars or four million network weights. Call fitness $f(\theta)$ : the score after running the policy, usually total episode return. Call $N$ the population size, here $N = 4$ . Call $k$ the survivors you breed from, here $k = 2$ . Call $\sigma$ the mutation spread, here $\sigma = 0.1$ , the standard deviation of the Gaussian noise you add.

One generation runs five steps:

Start with $N$ weight vectors.
Run each one for one or more full episodes and record $f$ .
Select the top $k$ scorers and drop the rest.
Vary the survivors by crossover, often a plain average, plus mutation, usually fresh draws from $\mathcal{N}(0, \sigma^2)$ added to each weight.
Replace the old population with the children, keep the single best parent unchanged, and repeat.

No gradient appears anywhere. The only signal is the ranking of scores.

Three flavors you will meet

Evolution Strategies keeps a mean vector $m$ and samples children as $m + \sigma \epsilon$ , with $\epsilon$ drawn from a standard Gaussian. It then moves $m$ toward the noise directions that scored well. Shared random seeds let thousands of workers generate the same perturbations without passing big weight vectors around, which is why OpenAI's 2017 work made this form popular for RL.

CMA-ES keeps the sampling idea but also learns the covariance $C$ of the noise, so sampling becomes $m + \sigma \mathcal{N}(0, C)$ . When one direction keeps paying off, the distribution stretches along it. That helps on stretched, ill-conditioned problems where a round ball wastes samples, at the price of more bookkeeping per step.

NEAT evolves the network itself, not just its weights. It starts from small nets with no hidden units, adds nodes and links by mutation, tags every new gene with a historical marking so two nets can cross over sensibly, and groups similar topologies into species that share fitness, so a promising new shape survives long enough for its weights to catch up.

Worked example: selecting from four policies

Four one-weight policies with measured returns:

$w_1 = -0.4$ , $f_1 = 8$
$w_2 = 0.3$ , $f_2 = 22$
$w_3 = 0.9$ , $f_3 = 35$
$w_4 = 1.6$ , $f_4 = 15$

Total fitness is $8 + 22 + 35 + 15 = 80$ , so mean fitness is $80 / 4 = 20.0$ . Ranked best to worst: $w_3$ at 35, $w_2$ at 22, $w_4$ at 15, $w_1$ at 8. Truncation with $k = 2$ keeps $w_3$ and $w_2$ ; the cut sits at 22 and everything below it is gone.

Crossover averages the two parents:

$(0.9 + 0.3) / 2 = 1.2 / 2 = 0.6$

Now apply fixed mutation draws for teaching, all consistent with $\sigma = 0.1$ :

Crossover child: $0.6 + 0.12 = 0.72$
Copy of best parent: $0.9 - 0.07 = 0.83$
Copy of second parent: $0.3 + 0.03 = 0.33$

With elitism the next generation is $g_1 = 0.90$ (elite copy of $w_3$ ), $g_2 = 0.72$ (crossover plus mutation), $g_3 = 0.83$ (best parent mutated), $g_4 = 0.33$ (second parent mutated). All four cluster near the two winners, 0.3 and 0.9, which is exactly what one step should do. None of the children has a score yet: the next step must run four fresh episodes, which is why the diagram closes the loop and flags the cost at the bottom.

Code

A pure-Python transcription of the worked example. It ranks the population, breeds the top two, and prints the next generation.

1weights = [-0.4, 0.3, 0.9, 1.6]2fitness = [8.0, 22.0, 35.0, 15.0]3
4mean_fitness = sum(fitness) / len(fitness)5ranked = sorted(zip(weights, fitness), key=lambda pair: pair[1], reverse=True)6best, second = ranked[0][0], ranked[1][0]7child_mean = (best + second) / 2.08
9next_gen = [best, round(child_mean + 0.12, 2), round(best - 0.07, 2), round(second + 0.03, 2)]10
11print(mean_fitness)  # -> 20.012print((best, second))  # -> (0.9, 0.3)13print(child_mean)  # -> 0.614print(next_gen)  # -> [0.9, 0.72, 0.83, 0.33]

Watch Out For

Each step burns full episodes

The symptom is slow wall-clock progress. This page used 4 episodes for one update; real runs use hundreds or thousands of policies per generation, and each one needs full episodes before the update happens. If your simulator is slow or each episode runs minutes on hardware, one generation can take all day.

The fix is to match the tool to the reward. When rewards are dense and the policy is differentiable, prefer policy gradient methods, where each episode gives gradient information for every step. When you must evolve, parallelize rollouts across workers, share random seeds instead of weight vectors, and start with small populations before scaling up.

Everyone crowds the first winner

The symptom is a population that looks done after a few rounds but scores poorly. All weights sit near 0.9 in the example, variation shrinks toward zero, and no child ever tries far regions again. Greedy top-k selection plus tiny $\sigma$ causes this fast.

The fix is to protect diversity on purpose. Raise $\sigma$ early and shrink it later, use tournament selection instead of always taking the top two, keep an elite of one rather than half the population, and restart or inject fresh random members when the spread of fitness stays near zero for several rounds.

The Quick Version

Neuroevolution scores whole policies and breeds winners, so it works without differentiable losses or slope signals.
One generation selects the top scorers, mixes parents by averaging, then adds Gaussian noise with spread $\sigma$ .
The four-policy example scores 8, 22, 35 and 15 for mean 20.0 and keeps the policies at 0.9 and 0.3.
Crossover mean 0.6 plus fixed tweaks builds the next generation at 0.90, 0.72, 0.83 and 0.33 before rescoring.
Evolution fits sparse, deceptive, or non-differentiable rewards; gradients fit dense rewards with cheap differentiable policies.