Value-Based Methods in RL

Score every move by its predicted future payoff, then take the max. The policy falls out for free, no separate actor required.

Action A2 rises from 2.5 to 3.7 after a reward of 1.0 plus 0.9 times the next best value 5.0, flipping the greedy choice

Why Does This Exist?

Judging actions by immediate reward is short-sighted: a $+1$ now can strand you three steps later. Learning a policy directly from full episodes works but needs complete trajectories before any update, which means crippling variance on long horizons.

Value-based methods change the question from "what should I do?" to "how much future reward does each move promise?" Learn an action-value function $Q(s, a)$ , then act greedily: $\pi(s) = \arg\max_a Q(s, a)$ . No actor network, no trajectory waits. One transition updates one estimate through bootstrapping, which is why Q-learning, SARSA, and deep Q-networks all sit on this foundation.

The one idea you need first

A Markov decision process frames the world as states, actions, rewards, and a discount $\gamma$ that prices the future. Value methods live inside that frame: $Q(s, a)$ is the expected discounted return from doing $a$ in $s$ and behaving well after. Skim the MDP page for the tuple; this page is what you build on top of it.

Think of It Like This

A chess coach who scores positions, not moves

A bad chess coach hands you a book of memorized openings: play this exact sequence here. It works until move twelve, then you are on your own. A good coach teaches you to score any position: queen centered plus king safe is worth a lot, a hanging rook subtracts a lot.

With scoring learned, moves need no book. Look at every legal move, score the resulting position, take the max. The score compresses thousands of future lines into one number, and the greedy rule reads it back out. That is $Q(s, a)$ plus argmax.

Where the analogy stops: a coach's scores come from deep search that is actually correct. Learned Q-values bootstrap off each other, so early scores are guesses training guesses, and the max operator amplifies the loudest guess. The pitfall below is that amplification biting.

How It Actually Works

The optimal values $Q^\*$ satisfy the Bellman optimality equation: the value of $(s, a)$ is the immediate reward plus the discounted best value onward.

Q^\*(s, a) = \mathcal{R}(s, a) + \gamma \sum_{s'} \mathcal{P}(s' \mid s, a) \max_{a'} Q^\*(s', a')

Here $\mathcal{R}$ is expected immediate reward, $\mathcal{P}$ the transition model, and $\gamma \in [0, 1)$ the discount. Model-free agents never see $\mathcal{P}$ ; from each transition $(S_t, A_t, R_{t+1}, S_{t+1})$ they build a sampled target $y_t = R_{t+1} + \gamma \max_{a'} Q(S_{t+1}, a')$ and nudge the estimate toward it:

Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha\,[\,y_t - Q(S_t, A_t)\,]

The bracketed term is the TD error $\delta$ . During training, pure argmax never tries anything new, so behavior is usually $\epsilon$ -greedy: random move with probability $\epsilon$ , greedy otherwise.

Worked example

State $S_0$ , two actions, $\gamma = 0.9$ , $\alpha = 0.4$ . Current estimates: $Q(S_0, A_1) = 3.0$ , $Q(S_0, A_2) = 2.5$ . The agent explores $A_2$ , gets $R = 1.0$ , lands in $S_1$ where $Q(S_1, A_1) = 5.0$ and $Q(S_1, A_2) = 4.0$ .

Next-state max: $\max(5.0, 4.0) = 5.0$ . Target: $y = 1.0 + 0.9 \times 5.0 = 5.5$ . TD error: $\delta = 5.5 - 2.5 = 3.0$ . Update: $Q(S_0, A_2) = 2.5 + 0.4 \times 3.0 = 3.7$ .

One transition flipped the greedy choice at $S_0$ : $3.7 > 3.0$ , so $A_2$ now wins. That is bootstrapping in one line, an estimate trained on another estimate, and Q-learning and SARSA are the two classic ways to run this loop.

Code

This snippet replays the worked update on a tiny Q-table. Only the standard library and numpy are used.

1from typing import Dict2import numpy as np3
4
5def q_update(q: Dict[str, np.ndarray], s: str, a: int, r: float, s_next: str, gamma: float = 0.9, alpha: float = 0.4) -> float:6    """One Bellman optimality backup. Returns the TD error."""7    target = r + gamma * float(np.max(q[s_next]))8    td_error = target - float(q[s][a])9    q[s][a] += alpha * td_error10    return td_error11
12
13q: Dict[str, np.ndarray] = {"S0": np.array([3.0, 2.5]), "S1": np.array([5.0, 4.0])}14print(round(q_update(q, "S0", 1, 1.0, "S1"), 2))  # -> 3.015print(q["S0"].tolist())  # -> [3.0, 3.7]16print(int(np.argmax(q["S0"])) + 1)  # -> 2

The TD error of 3.0 lifts $A_2$ from 2.5 to 3.7, so argmax now returns action 2.

Watch Out For

Believing the max of noisy estimates

$\max_{a'} Q(S', a')$ grabs the loudest estimate, and noise has a loud side: $\mathbb{E}[\max] \ge \max(\mathbb{E})$ . With several similar-valued actions plus approximation noise, the target keeps sampling positive spikes and values inflate across dead-end states. Symptom: Q-values climb far past any return the agent ever observes, then the policy chases ghosts.

Split selection from evaluation as in double Q-learning: pick $a^\*$ with the online net, score it with the target net. If your values exceed plausible returns, suspect the max before the reward.

Decaying epsilon before values mean anything

Teams anneal $\epsilon$ to near zero in the first thousand steps to "lock in" learning, but early Q-values are initialization noise, so greed locks onto whichever action started highest. Symptom: the policy converges fast to something mediocre and never recovers, while a longer-exploring run passes it.

Keep $\epsilon$ high until TD errors stabilize, then decay slowly. Fast convergence to a bad policy is not convergence working; it is exploration ending early.

The Quick Version

Value-based methods learn $Q(s, a)$ , the expected discounted return of each move, and act by argmax with no actor network.
The Bellman optimality target $R + \gamma \max_{a'} Q(s', a')$ turns one transition into one update via the TD error.
Bootstrapping makes learning off-policy and sample-efficient: estimates train on estimates.
The max operator overestimates under noise, so split action selection from evaluation with double Q-learning.
Epsilon-greedy must stay open long enough for values to earn their greed, or the policy locks onto noise.