Fundamentals of Reinforcement Learning

An agent tries actions, banks rewards, and learns a policy that maximizes long term return. Delayed rewards make credit assignment the core problem.

An agent loops state to action to reward, and with rewards 0, minus 2 and plus 10 at gamma 0.9 the return from the start is 6.3

Why Does This Exist?

Supervised learning needs a teacher who names the right answer for every input. Sequential control rarely has one. No one hands a chess program the best move for each board, or a walking robot the exact motor voltage for each millisecond.

Three gaps appear once actions have consequences over time. Feedback is evaluative, not instructive: the world returns a score for the move you took, never the move you should have taken. Consequences are delayed: a move at step 0 can decide a win or a pileup at step 50. Data is active: your current policy picks which states you will see next, so early mistakes push you into unfamiliar ground.

Reinforcement learning names this loop and optimizes it. The details of states and transitions live in Markov decision processes. What a state is worth over the long run lives in value functions. This page covers the loop itself, the return it maximizes, and the credit problem that makes the loop hard.

The one idea you need first

A policy is the agent behavior, written $\pi(a \mid s)$ , the probability of taking action $a$ in state $s$ . A trajectory is one run of experience: states, actions and rewards in order. Everything else here scores trajectories, not single moves.

Think of It Like This

Training a puppy with timed treats

You teach a puppy to sit. You cannot explain leg angles. You say the cue, the puppy tries things, and the instant its weight drops you give a treat. That treat is $R_{t+1}$ , the state is the pose plus your cue, the action is what the puppy just did.

Timing is the whole lesson. Treat twenty seconds late, after barking and carpet chewing, and the puppy credits the wrong move. That is temporal credit assignment: which past action earned a later reward. Real agents have the same issue without fur or hunger. They maximize the exact number you defined, including its loopholes.

Where it stops: puppies bring instinct and fatigue. The math agent brings neither, only the sum you told it to maximize.

How It Actually Works

An agent and an environment alternate at steps $t = 0, 1, 2, \dots$ . At $S_t$ the agent draws $A_t$ from $\pi$ . The world replies with $R_{t+1}$ and $S_{t+1}$ under dynamics $p(s', r \mid s, a)$ . In words: $S_t$ is where you are, $A_t$ is what you try, $R_{t+1}$ is the score, $S_{t+1}$ is where you land.

The goal is never one reward. It is the discounted return from $t$ onward, with discount $\gamma$ in $[0, 1)$ :

G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \dots

In words: add future rewards, shrink each step by another $\gamma$ . Recursion helps: $G_t = R_{t+1} + \gamma G_{t+1}$ . At $\gamma = 0$ the agent is myopic and only sees $R_{t+1}$ . Near 1 it weighs far rewards heavily, but the sum still converges because $\gamma < 1$ .

Worked example

Use $\gamma = 0.90$ and one short episode. Start, corridor, goal approach, then goal:

$t = 0$ : $R_1 = 0.0$
$t = 1$ : $R_2 = -2.0$ , rough ground cost
$t = 2$ : $R_3 = +10.0$ , goal

Work backward. $G_2 = 10.00$ . Then $G_1 = -2.00 + 0.90 \times 10.00 = 7.00$ . Then $G_0 = 0.00 + 0.90 \times 7.00 = 6.30$ . Direct sum checks: $0.0 + 0.90 \times (-2.0) + 0.81 \times 10.0 = -1.80 + 8.10 = 6.30$ .

Contrast a greedy grab: reward $+1.50$ now, then $-5.00$ , then $0.00$ . Its return is $1.50 + 0.90 \times (-5.00) = -3.00$ . Ahead after one step, far behind over the episode. That gap is why the objective is $G_t$ and not $R_{t+1}$ .

Code

This snippet computes the same backward return, so you can check any reward list:

1def discounted_returns(rewards: list[float], gamma: float) -> list[float]:2    """Return G_t for each t using G_t = R_{t+1} + gamma * G_{t+1}."""3    out: list[float] = [0.0] * len(rewards)4    running = 0.05    for t in range(len(rewards) - 1, -1, -1):6        running = rewards[t] + gamma * running7        out[t] = round(running, 2)8    return out9
10
11rewards = [0.0, -2.0, 10.0]12print(discounted_returns(rewards, 0.9))  # -> [6.3, 7.0, 10.0]13print(discounted_returns([1.5, -5.0, 0.0], 0.9))  # -> [-3.0, -5.0, 0.0]

The first list matches the hand calculation $[6.30, 7.00, 10.00]$ . The second shows the greedy trap at $-3.00$ .

Watch Out For

Chasing the immediate reward

Newcomers from supervised learning optimize $R_{t+1}$ by habit. The symptom is a policy that grabs small early points and stalls before a larger late payoff, exactly the $+1.50$ versus $+6.30$ split above.

Fix it in the objective, not the tuning. Train on $G_t$ or on value estimates from value functions, and keep a baseline that forces the agent to compare full episode scores during exploration versus exploitation.

Setting gamma without thinking

At $\gamma = 0$ the agent cannot see past one step. At $\gamma = 1$ in a continuing task the sum can diverge and tiny reward misspecifications compound forever. The symptom is either dithering near small rewards or unstable values that keep growing.

Pick $\gamma$ from the horizon you care about. Use around 0.9 to 0.99 for delayed goals, lower only when only near term behavior matters, and always keep it below 1 for continuing tasks.

The Quick Version

The loop is $S_t$ to $A_t$ to $R_{t+1}$ and $S_{t+1}$ , repeated, with data generated by the policy itself.
The objective is expected discounted return $G_t$ , not immediate reward.
Discount $\gamma$ sets the horizon: 0 is myopic, near 1 values far payoffs.
With rewards 0, minus 2 and plus 10 at gamma 0.9, the start return is 6.3 against minus 3.0 for the greedy grab.
Value functions carry delayed credit backward so each step can act forward.