Goal-Conditioned Reinforcement Learning

One policy learns many goals at once by taking the target as input. The same network walks to any destination instead of retraining per address.

One goal conditioned value function scores state goal pairs, and at gamma 0.9 north gives minus 5.22 while south gives minus 6.13

Why Does This Exist?

A fixed policy bakes in one destination. Move the target and it fails, even when the physics did not change. Training one network per goal ignores shared structure and scales badly.

Goal conditioned RL feeds the goal $g$ into policy $\pi(a \mid s, g)$ and value $Q(s, a, g)$ , after Kaelbling and UVFA work by Schaul and coauthors. One net then serves many tasks, failures can be relabeled as wins for the place actually reached in hindsight experience replay, and higher levels in hierarchical RL can command subgoals through the same interface.

The one idea you need first

A goal space $\mathcal{G}$ lists possible targets, often a subset of states via map $\phi(s)$ . Reward is usually sparse: $0$ when close enough to $g$ , $-1$ otherwise. Sparse keeps shortest path incentives clean where dense distance can trap the agent behind walls.

Think of It Like This

A GPS unit, not a fixed commute

A fixed commuter only knows office to home. Ask for the airport and it blanks. A GPS takes current position plus destination and gives turn by turn moves for any pair.

The value $V(s, g)$ is travel time between any two pins. Miss an exit and it recomputes from the new state. Reach the wrong lot and it still learned that route, which hindsight later reuses.

Where it stops: a GPS starts with a surveyed map. The agent starts with none and learns distances by trial.

How It Actually Works

Inputs are state $s$ and goal $g$ . Two builds exist. Joint concat runs $[s; g; a]$ through one MLP for $Q_{\theta}$ . Factored build encodes $\phi(s)$ and $\psi(g)$ separately and dots them for $V$ , which helps lookup across many goals.

Bellman holds per goal:

Q^*(s, a, g) = R(s, a, g) + \gamma \max_{a'} Q^*(s', a', g)

In words: immediate goal reward plus discounted best future value for the same $g$ . Replay stores $(s, a, s', g)$ and minimizes TD error across sampled goals.

Worked example

Stand at $s = [1.0, 1.0]$ , target $g = [4.0, 4.0]$ , $\gamma = 0.90$ , step penalty $-1.0$ , terminal $0.0$ . North goes to $[1.0, 2.0]$ with $6$ steps left; south goes to $[1.0, 0.0]$ with $8$ steps left.

Optimal remaining value for $D$ steps of $-1$ is $-(1 - \gamma^D)/(1 - \gamma)$ . So $\gamma^6 = 0.531441$ gives $V_N = -(1 - 0.531441)/0.10 = -4.6856$ . And $\gamma^8 = 0.430467$ gives $V_S = -(1 - 0.430467)/0.10 = -5.6953$ .

Bellman Q adds one penalty step:

$Q_N = -1.00 + 0.90 \times (-4.6856) = -5.217$
$Q_S = -1.00 + 0.90 \times (-5.6953) = -6.126$

Since $-5.217$ beats $-6.126$ , the policy picks north toward the goal.

Code

Same choice in code:

1import numpy as np2
3
4def remaining_value(steps: int, gamma: float = 0.90) -> float:5    """Discounted sum of -1 for D steps left."""6    if steps <= 0:7        return 0.08    return float(-(1.0 - gamma**steps) / (1.0 - gamma))9
10
11def q_value(steps_left: int, gamma: float = 0.90) -> float:12    """One -1 step plus gamma times remaining value."""13    return float(-1.0 + gamma * remaining_value(steps_left, gamma))14
15
16print(round(remaining_value(6), 4))  # -> -4.685617print(round(remaining_value(8), 4))  # -> -5.695318print(round(q_value(6), 3))  # -> -5.21719print(round(q_value(8), 3))  # -> -6.126

Outputs match the hand math, and the larger Q picks north.

Watch Out For

Swapping sparse reward for straight line distance

Dense $-\lVert s - g \rVert$ looks helpful but punishes detours. Behind a U wall the shortest Euclidean step pushes into the barrier while the true path first moves away. The symptom is a policy stuck pressing the wall.

Stay sparse with $-1$ plus hindsight relabeling. If shaping is needed, learn a reachability distance rather than raw Cartesian gap.

The Quick Version

Policy and values take $(s, g)$ , so one net serves many destinations.
UVFA builds are joint MLP or factored dot product of state and goal codes.
Sparse $0$ versus $-1$ encodes shortest discounted path without wall traps.
At gamma 0.9, north gives Q minus 5.22 and south gives minus 6.13, so north wins.
Failed runs relabel to reached goals, which powers hindsight replay.