Basic Concepts in Reinforcement Learning

An agent tries actions, gets rewards, and learns a policy that maximizes total future reward. This page names the five moving parts and shows how discounting weighs the future.

The agent environment loop turns states into actions and rewards, and a four-step return with discount 0.9 totals 8.39

Why Does This Exist?

Supervised learning needs a correct label for every input. Control problems rarely have one. A walking robot, a game player, or a warehouse scheduler only gets an occasional score after a long sequence of moves, and no one tells it which of the hundred past decisions deserves the credit.

Three things make that hard. First, rewards arrive late, so early actions must be judged by consequences many steps away. Second, every action changes which situations the agent will see next, so the data distribution moves as the agent learns. Third, feedback is evaluative rather than instructive: a reward says how well things went, never which move would have been best.

Reinforcement learning is the framework for exactly this setup. This page defines the loop and the objective. The formal state transition model lives in MDP foundations, the functions that score states and actions live in value functions, and the tension between trying new moves and repeating good ones lives in exploration versus exploitation.

You need one prerequisite idea first. An expectation is a weighted average over randomness. When transitions or policies are stochastic, every return and value below is an expectation over the trajectories the policy produces.

Think of It Like This

A dog trainer with a clicker and treats

You train a dog to run a small obstacle course. You never push the dog through the motions. You watch, you click and treat good moments, you withhold treats for bad ones, and the dog adjusts.

The course layout at each moment is the state. The dog's next move is the action. The click and treat is the reward. The dog's habits are the policy. The total treats over a full run, with earlier treats mattering slightly more than later ones, is the return.

Where the analogy stops: the dog generalizes from a handful of runs and reads your body language. An RL agent needs thousands of episodes, an explicit discount number, and a stored value estimate to connect a treat now with a turn taken ten seconds ago.

How It Actually Works

At each step $t = 0, 1, 2, \dots$ the agent and environment exchange three signals:

State $S_t$ in a state set $\mathcal{S}$ : everything the agent is allowed to see about the situation.
Action $A_t$ in an action set $\mathcal{A}(S_t)$ : the choice the agent makes in that state.
Reward $R_{t+1}$ , a single number: the environment's score for the transition that just happened.

Two functions govern the loop:

Transition dynamics $P(s', r \mid s, a)$ : the probability of landing in next state $s'$ with reward $r$ after taking action $a$ in state $s$ . This is the world's physics, not the agent's choice.
Policy $\pi(a \mid s)$ : the probability of picking action $a$ in state $s$ . A deterministic policy writes $a = \mu(s)$ . The policy is the only thing the agent controls, and learning means improving it.

One full interaction is a trajectory $\tau = (S_0, A_0, R_1, S_1, A_1, R_2, \dots)$ . Tasks come in two shapes. An episodic task ends at a terminal state $S_T$ , like checkmate or reaching a goal. A continuing task runs forever, like balancing or server cooling, with $T = \infty$ .

Return and the discount factor

The agent does not maximize one reward. It maximizes the discounted return from step $t$ onward, with discount factor $\gamma$ in $[0, 1)$ :

$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \dots$

Equivalently, $G_t = R_{t+1} + \gamma G_{t+1}$ . Here $R_{t+k+1}$ is the reward $k+1$ steps in the future and $\gamma^k$ shrinks its weight. The factor $\gamma$ does two jobs. It bounds the sum for continuing tasks, since rewards bounded by $R_{\max}$ give $G_t \le R_{\max} / (1 - \gamma)$ . And it sets the horizon: $\gamma = 0$ is fully myopic and only the next reward matters, while $\gamma$ near 1 makes the agent farsighted. A handy rule is that the effective horizon is about $1 / (1 - \gamma)$ steps.

Because the future is random, values are expectations of this return. The state value $V^\pi(s) = \mathbb{E}_\pi[G_t \mid S_t = s]$ scores a state under a policy, and the action value $Q^\pi(s, a) = \mathbb{E}_\pi[G_t \mid S_t = s, A_t = a]$ scores one move followed by that policy. The goal of RL is an optimal policy $\pi^*$ maximizing $\mathbb{E}[G_0]$ from the start.

Worked example

Four rewards arrive in order: $R_1 = 2.0$ , $R_2 = -1.0$ , $R_3 = 0.0$ , $R_4 = 10.0$ . The episode ends after step 4, so $G_4 = 0$ . Fix $\gamma = 0.9$ and work backward with $G_t = R_{t+1} + 0.9 \, G_{t+1}$ :

$G_3 = 10.0 + 0.9 \times 0.0 = 10.0$
$G_2 = 0.0 + 0.9 \times 10.0 = 9.0$
$G_1 = -1.0 + 0.9 \times 9.0 = -1.0 + 8.1 = 7.1$
$G_0 = 2.0 + 0.9 \times 7.1 = 2.0 + 6.39 = 8.39$

Direct expansion agrees: $G_0 = 2.0 + 0.9 \times (-1.0) + 0.9^2 \times (0.0) + 0.9^3 \times (10.0) = 2.0 - 0.90 + 0.0 + 7.29 = 8.39$ . The same trajectory under $\gamma = 0$ gives $G_0 = 2.00$ , under $\gamma = 0.5$ gives $2.75$ , and undiscounted gives $11.00$ . The diagram draws these four rewards and the backward pass that lands on 8.39.

Code

The snippet computes the same backward pass for several discount values, so you can check the hand arithmetic above.

1def discounted_returns(rewards: list[float], gamma: float) -> list[float]:2    """Backward discounted returns G_t = R_{t+1} + gamma * G_{t+1}."""3    out: list[float] = [0.0] * len(rewards)4    running = 0.05    for t in range(len(rewards) - 1, -1, -1):6        running = rewards[t] + gamma * running7        out[t] = running8    return out9
10
11if __name__ == "__main__":12    rewards = [2.0, -1.0, 0.0, 10.0]13    for gamma in [0.0, 0.5, 0.9, 1.0]:14        print(gamma, [round(v, 2) for v in discounted_returns(rewards, gamma)])

1# -> expected output:20.0 [2.0, -1.0, 0.0, 10.0]30.5 [2.75, 1.5, 5.0, 10.0]40.9 [8.39, 7.1, 9.0, 10.0]51.0 [11.0, 9.0, 10.0, 10.0]

Watch Out For

Steering by the latest reward instead of the return

The symptom: the agent grabs small immediate rewards and walks past the goal, or refuses a costly move that unlocks a big payoff later. In the example above, a greedy agent at step 1 sees $-1.0$ and backs away, never reaching the $+10.0$ two steps later.

The fix: never select actions from $R_{t+1}$ alone. Score states and moves with $V(s)$ or $Q(s, a)$ , which estimate expected discounted return, and set $\gamma$ large enough that the effective horizon covers the delay between cause and payoff.

Using gamma equal to 1.0 on a task that never ends

The symptom: on a continuing task the return sum diverges, value targets grow without bound, and training destabilizes into overflow or oscillation. It looks like a learning rate bug but no learning rate fixes it.

The fix: keep $\gamma$ strictly below 1 for continuing tasks, commonly 0.95 to 0.999, so returns stay bounded by $R_{\max} / (1 - \gamma)$ . If the task truly needs undiscounted optimization, switch the objective to average reward rather than forcing $\gamma = 1$ .

The Quick Version

An RL loop exchanges state, action, and scalar reward, with transition dynamics owned by the world and the policy owned by the agent.
The objective is expected discounted return $G_t = R_{t+1} + \gamma G_{t+1}$ , not the latest reward.
The discount factor bounds infinite sums and sets the planning horizon near $1 / (1 - \gamma)$ .
Episodic tasks end in a terminal state; continuing tasks run forever and require $\gamma$ below 1.
In the worked example, rewards 2.0, minus 1.0, 0.0, and 10.0 with discount 0.9 give a total return of 8.39.