Reward Function

Why Does This Exist?

No one tells a reinforcement learning agent which action was correct. It tries, watches the world change, and gets one number back. Without that number there is no learning signal at all, only random wandering through thousands of steps that all look the same.

The reward function is that number. It maps a single transition to a scalar, and everything else in RL is commentary on those scalars. A delivery robot that reaches the charger in 40 steps gets 40 numbers. If the numbers reward arrival and punish dawdling, the robot learns short routes. If they reward wheel speed, it learns to spin in place very fast.

This page covers the mapping itself: what it takes in, what it returns, and how a short chain of rewards adds to a return you can check. The rulebook around it lives in MDP foundations. What those rewards predict over the long run lives in value functions. How to add helpful hints without changing the goal lives in reward shaping.

The one idea you need first is the return. Write the rewards after time $t$ as $R_{t+1}, R_{t+2}, \dots$ and pick a discount $\gamma$ (gamma) between 0 and 1. The return is $G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \dots$. Here each $R$ is one immediate reward and $\gamma$ says how much you care about later rewards relative to now. A $\gamma$ of 0.9 keeps the future relevant without letting far away steps dominate.

Think of It Like This

How It Actually Works

The general form takes the full transition. With $s$ the current state, $a$ the action, and $s^\prime$ the next state:

$R(s, a, s^\prime) \in \mathbb{R}$

In plain words: $s$ is where you are, $a$ is what you do, $s^\prime$ is where you land, and $R$ is the single score for that triple. Simpler forms exist. $R(s, a)$ averages over possible landings, and $R(s^\prime)$ scores only the landing zone, such as plus 10 inside the goal and minus 1 elsewhere. All three are the same idea with fewer arguments.

The reward hypothesis behind all of RL says every goal can be expressed as maximizing the expected discounted sum of these scalars. That is a strong claim and it explains why reward design feels so high stakes. You do not get to tell the agent to be careful, quick, and gentle in separate channels. You get one number per step, so careful, quick, and gentle must all live inside it.

Sparse versus dense

A sparse reward pays only at the goal, such as plus 1 on arrival and 0 elsewhere. It never lies but it rarely speaks, so random exploration may never find it in a large room. A dense reward speaks every step, such as minus 1 per step plus 10 at the goal. It teaches fast but it can be gamed, because any step signal is a proxy and proxies have loopholes. The practical rule: start with the true sparse goal, then add dense terms you can defend, and test that the fastest way to earn is still the task you wanted.

Worked example

A chain has states $s_0$ to $s_3$ plus a goal. The reward is minus 1.0 on every non goal step and plus 10.0 on entering the goal. Discount $\gamma$ is 0.9. The direct path takes four steps:

$s_0 \to s_1 \to s_2 \to s_3 \to s_{goal}$

Rewards are $R_1 = -1.0$, $R_2 = -1.0$, $R_3 = -1.0$, $R_4 = 10.0$. The return from the start is:

$G_0 = R_1 + \gamma R_2 + \gamma^2 R_3 + \gamma^3 R_4$

$G_0 = -1.0 + 0.9 \times (-1.0) + 0.81 \times (-1.0) + 0.729 \times 10.0$

$G_0 = -1.0 - 0.90 - 0.81 + 7.29 = 4.58$

That 4.58 is the number the diagram draws. A five step wander that stalls once earns 3.12 under the same rule, so the step penalty plus discount ranks the short path first by 1.46. If a designer instead pays plus 2.0 for every right move and 0.0 for every left move, a two state loop earns 10.53 forever without ever reaching the goal, which beats 4.58 and teaches the agent to loop. That contrast is why the dense terms need a step cost.

Code

This snippet computes the same 4.58 from the same four rewards, so you can check the arithmetic in one run:

1from typing import List2
3
4def discounted_return(rewards: List[float], gamma: float = 0.9) -> float:5    """Return G0 for one reward list. rewards[0] is R1."""6    total = 0.07    for r in reversed(rewards):8        total = r + gamma * total9    return total10
11
12if __name__ == "__main__":13    direct = [-1.0, -1.0, -1.0, 10.0]14    print(round(discounted_return(direct, gamma=0.9), 3))15    # -> 4.5816    stall = [-1.0, -1.0, -1.0, -1.0, 10.0]17    print(round(discounted_return(stall, gamma=0.9), 3))18    # -> 3.122

The second line is the stalling path at 3.122. Same rule, one extra minus one step, lower return. That gap is the step penalty doing its job.

Watch Out For

The Quick Version

• The reward function maps one transition $(s, a, s^\prime)$ to one scalar score.
• The agent never sees the goal directly. It sees the discounted sum $G_t$ of those scalars.
• Sparse rewards tell the truth rarely. Dense rewards teach fast and can be gamed.
• Four steps of minus 1, minus 1, minus 1, plus 10 at gamma 0.9 return 4.58.
• Charge steps, bonus the goal, and test on goal rate rather than shaped return.
• Keep step rewards near unit scale so the optimizer sees stable updates.

What to Read Next

• Markov Decision Processes
• Value Functions
• Reward Shaping