Hindsight Experience Replay (HER)

Why Does This Exist?

Goal tasks pay sparsely by nature: a robot gets 0 for placing the block right and minus 1 everywhere else. Random flailing almost never lands a distant target, so the buffer fills with pure failure and value updates have no nonzero signal to climb. Everything looks equally futile.

Hand shaping the reward, like paying negative distance, seems to help but bends the goal. Agents learn to hover near obstacles or freeze at local optima instead of finishing the job. Hindsight replay, from Andrychowicz and coauthors in 2017, takes another road. Every failed run at goal g is a perfect run at wherever the agent ended up, so it copies each transition under the achieved goal and recomputes the reward. Failures become successes without touching the true reward function. It slots into goal-conditioned RL, builds on plain experience replay, and replaces much of reward shaping.

The one idea you need first

A goal-conditioned value Q(s, a, g) answers many questions at once: how good is this move for each goal I might want. The sparse reward is 0 when the next state's achieved goal lands within tolerance eps of g, else minus 1. One network then serves every goal, which is exactly what relabeling trains.

Think of It Like This

How It Actually Works

States pair with goals through a mapping m from state to achieved goal, and the policy plus values read both: pi(a | s, g) and Q(s, a, g). A rollout at goal g stores the usual tuples (s, a, r, s_next, g), mostly with r = -1.

Hindsight copies each transition under replacement goals g' taken from visited states, then recomputes r' = r(s_next, g'). The tested strategies are final (relabel with the last state only), future (sample a few states from later in the same episode, the winner at about four per step), episode (sample anywhere in the episode), and random (sample old history). future wins because its goals stayed reachable within the remaining steps.

This only works off policy. Changing the goal after the fact breaks the on policy promise that actions came from the policy under that goal, while off policy methods like DDPG or SAC just evaluate the Bellman backup under g':

The transition itself stays valid because physics ignores the goal label.

Worked example

A line task starts at 0.0 with target g = 10.0, tolerance 0.5, discount 0.90, and rate 0.20. Three steps of +2.0 visit 2.0, 4.0, 6.0, every distance above 0.5, so the buffer holds three minus 1 transitions and learning stalls.

Relabel with the final strategy at g' = 6.0. The last step now ends exactly on its goal: reward r' = 0.0, terminal value 0.0, target y = 0.0. From Q = -2.50 the error is +2.50 and the new value is -2.50 + 0.20 * 2.50 = -2.00.

The middle step still misses (|4.0 - 6.0| = 2.0, reward minus 1) but bootstraps from the improved downstream value: y = -1.0 + 0.90 * (-2.00) = -2.80. From Q = -4.00 the error is +1.20 and the new value is -4.00 + 0.20 * 1.20 = -3.76. One manufactured success backed signal two steps deep, and many endpoints tile the goal space from there.

Code

This snippet replays the relabeling with plain Python, printing the same reward flip and Q updates derived by hand.

1def sparse_reward(next_state: float, goal: float, tol: float = 0.5) -> float:2    return 0.0 if abs(next_state - goal) <= tol else -1.03
4
5states = [0.0, 2.0, 4.0, 6.0]6target, hindsight = 10.0, 6.07orig = [sparse_reward(s_next, target) for s_next in states[1:]]8relabelled = [sparse_reward(s_next, hindsight) for s_next in states[1:]]9print(orig)  # -> [-1.0, -1.0, -1.0]10print(relabelled)  # -> [-1.0, -1.0, 0.0]11
12gamma, alpha = 0.90, 0.2013q_last = -2.50 + alpha * ((0.0 + gamma * 0.0) - (-2.50))14q_mid = -4.00 + alpha * ((-1.0 + gamma * q_last) - (-4.00))15print(f"{q_last:.2f} {q_mid:.2f}")  # -> -2.00 -3.76

Rewards flip from all minus 1 to [-1.0, -1.0, 0.0], and Q values land on -2.00 and -3.76 exactly as the hand pass computed.

Watch Out For

The Quick Version

• Sparse goal rewards starve learning because nearly every rollout scores minus 1 on every step.
• Relabeling files each transition under achieved goals and recomputes the reward, manufacturing successes from failures.
• The future strategy with about four later states per step beats final, episode, and random sampling.
• Hindsight needs off policy learners, since the goal swap breaks the on policy sampling promise while physics stays valid.
• The worked run turned goal ten failure into goal six success, lifting Q from minus 2.50 to minus 2.00 and then minus 3.76 one step back.
• Read goal-conditioned RL for the framing, experience replay for the buffer, and reward shaping for what this replaces.

What to Read Next

• Goal-Conditioned Reinforcement Learning
• Experience Replay
• Reward Shaping