Q-Learning vs SARSA in RL

Q-learning learns the value of the best action you could take next, while SARSA learns the value of the action you actually take, exploration mistakes included. That one-word difference decides whether your agent hugs the cliff edge or walks around it.

Q-learning backs up the best next value while SARSA backs up the action actually taken, so the same cliff step teaches 4.0 on one side and minus 45.5 on the other

Why Does This Exist?

Temporal difference control learns action values straight from experience, with no model of the environment. Both algorithms you meet here do that. They split on one question: when you update the value of what you just did, do you assume the next step goes perfectly, or do you price in your own exploration mistakes?

Q-learning (Watkins, 1989) assumes perfection. SARSA takes the action its current policy actually picks, stumbles included. Sutton and Barto's cliff walking grid makes the split unforgettable: every step pays $-1$ , stepping off the cliff pays $-100$ and returns you to the start. Q-learning learns the short path along the cliff edge, then keeps falling off while it explores. SARSA learns the longer path one row back and stays dry.

This page covers the two update rules and one worked transition that shows them disagreeing. For the Bellman equations they both sample, see Bellman equations. For learning from full episodes instead of one-step bootstrapping, see temporal difference learning.

You need one prerequisite: an $\epsilon$ -greedy policy picks the currently best action with probability $1 - \epsilon$ and a random action with probability $\epsilon$ . That random fraction is the whole story below.

Think of It Like This

Two hiking guides above a cliff path

You hire two guides to learn the fastest safe route along a coastal cliff.

The first guide plans like nothing ever goes wrong. "The edge path takes ten minutes," she says, timing a perfect walk on a calm day. She never factors in that her own boots slip sometimes. Her report describes the optimal walk, not the walk her group actually survives.

The second guide walks with the group, slips included. When a hiker stumbles near the edge, she feels the scare and updates her notes: this path costs more than the stopwatch says, because her group does not walk it perfectly. Her report describes the walk her group actually produces, and she reroutes one row inland.

Map it point for point: the stopwatch is the $\max$ in Q-learning, the stumble is the exploratory action SARSA samples, and the inland reroute is the safer policy SARSA converges to under fixed exploration.

Where it stops: guides choose caution on purpose, while SARSA is not brave or careful. It simply backs up whatever action the behavior policy sampled. Shrink $\epsilon$ toward zero and both guides recommend the same edge path.

How It Actually Works

Both methods keep a table $Q(s, a)$ , the estimated value of taking action $a$ in state $s$ . Both see a transition $(S_t, A_t, R_{t+1}, S_{t+1})$ , both scale the correction by a step size $\alpha$ in $(0, 1]$ , and both discount the future by $\gamma$ . They differ in exactly one term: the bootstrap target.

The two targets, side by side

Q-learning asks what the greedy policy would get from the next state, ignoring which action was actually picked:

Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha \left[ R_{t+1} + \gamma \max_{a} Q(S_{t+1}, a) - Q(S_t, A_t) \right]

SARSA asks what the behavior policy actually got, because it backs up the sampled next action $A_{t+1}$ :

Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha \left[ R_{t+1} + \gamma Q(S_{t+1}, A_{t+1}) - Q(S_t, A_t) \right]

The name SARSA is the tuple it consumes: state, action, reward, next state, next action. Because the target uses the action the $\epsilon$ -greedy policy sampled, SARSA evaluates the policy it runs, which makes it on-policy. Because Q-learning's target never mentions the sampled next action, its behavior policy can be anything exploratory while it learns the greedy policy's values, which makes it off-policy.

Worked example: one transition, two answers

Your agent stands at $S_0$ , takes the edge action, lands in $S_1$ , and collects $R = -1.0$ . In $S_1$ there are two actions: stay on the path worth $Q(S_1, a_1) = 10.0$ , or fall off the cliff worth $Q(S_1, a_2) = -100.0$ . Set $\alpha = 0.5$ , $\gamma = 0.9$ , and $Q(S_0, A_{\text{edge}}) = 0.0$ . Exploration misfires and the policy samples the fatal action, so $A_{t+1} = a_2$ .

Q-learning ignores the misfire and takes the maximum:

\max_{a} Q(S_1, a) = \max(10.0, -100.0) = 10.0

y_Q = -1.0 + 0.9 \times 10.0 = 8.0, \quad Q(S_0, A_{\text{edge}}) \leftarrow 0.0 + 0.5 \times (8.0 - 0.0) = 4.0

SARSA backs up the action actually taken:

Q(S_1, A_{t+1}) = Q(S_1, a_2) = -100.0

y_S = -1.0 + 0.9 \times (-100.0) = -91.0, \quad Q(S_0, A_{\text{edge}}) \leftarrow 0.0 + 0.5 \times (-91.0 - 0.0) = -45.5

Same transition, opposite lessons. Q-learning rewards the edge action with $+4.0$ because a perfect agent survives it. SARSA punishes it with $-45.5$ because its agent just fell off. The diagram above draws exactly these numbers.

When the two answers meet

If $\epsilon$ decays toward zero, the behavior policy stops sampling fatal actions, SARSA's target converges to the maximum, and both methods approach the optimal values. That is why the cliff gap is an online-performance story, not an asymptotic one. For the middle ground that averages over the behavior policy instead of sampling one action, read Expected SARSA. For what the $\max$ does to noisy estimates, read maximization bias and Double Q-learning.

Code

This snippet replays the worked example so you can feel the one-term difference. Pure standard library, nothing to install:

1def q_learning_update(q: float, r: float, next_qs: list[float], alpha: float = 0.5, gamma: float = 0.9) -> float:2    target = r + gamma * max(next_qs)  # greedy continuation, sampled action ignored3    return q + alpha * (target - q)4
5
6def sarsa_update(q: float, r: float, next_q_taken: float, alpha: float = 0.5, gamma: float = 0.9) -> float:7    target = r + gamma * next_q_taken  # the action actually sampled8    return q + alpha * (target - q)9
10
11next_qs = [10.0, -100.0]  # stay on path, fall off cliff12taken = next_qs[1]        # exploration misfires and samples the fall13
14print(q_learning_update(0.0, -1.0, next_qs))  # -> 4.015print(sarsa_update(0.0, -1.0, taken))         # -> -45.5

The prints match the hand computation: $4.0$ for the rule that assumes perfect play, $-45.5$ for the rule that prices in the fall.

Watch Out For

Training Q-learning live where mistakes cost money

Q-learning's indifference to exploration noise is safe in a simulator and dangerous on hardware. A robot learning online still takes random actions during training, and Q-learning will keep recommending the cliff edge because its values assume those random actions never happen. The symptom is a great greedy policy with terrible training returns: broken hardware, scratched floors, blown episodes. If mistakes cost more than simulation time, train with SARSA or decay $\epsilon$ fast, and only switch to Q-learning's answer once exploration is nearly off.

Reading SARSA's values as optimal values

With $\epsilon$ fixed above zero, SARSA converges to the values of the exploratory policy, not the optimal one. Teams then deploy the greedy policy and wonder why returns jump above what the table predicted: the table priced in random actions that deployment never takes. The fix is bookkeeping, not a new algorithm. Treat SARSA's table as the value of training behavior, decay $\epsilon$ before your final evaluation, and compare greedy-to-greedy when you report results.

The Quick Version

Q-learning targets $R + \gamma \max_a Q(S', a)$ , learning optimal values while any exploratory policy collects the data.
SARSA targets $R + \gamma Q(S', A')$ with the sampled next action, learning the value of the exploratory policy it actually runs.
On one cliff transition with reward $-1$ , the two targets are $8.0$ and $-91.0$ , giving updates of $4.0$ and $-45.5$ from zero.
During training with fixed exploration, Q-learning walks the risky edge while SARSA detours inland and earns steadier returns.
Decaying exploration to zero closes the gap, and both methods converge to the optimal policy.