Policy in Reinforcement Learning

A policy is the agent rule for picking actions, either one fixed move per state or a probability mix that keeps exploring.

Q values 2.40, 1.10, and 3.20 become choice chances 0.29, 0.08, and 0.64 at temperature 1 while greedy picks the top move

Why Does This Exist?

An agent that cannot pick actions cannot act. Value functions score situations, models predict next states, but neither says what to do now. The policy is the part that decides.

Without a clear policy form you lose three things. You cannot separate scoring from acting, so Q values never turn into behavior. You cannot tune exploration, so the agent either repeats one move or flails at random. And you cannot run policy gradients, because there are no policy weights to climb.

This page covers the two policy forms, how values turn into choices, and what optimal means. For the world they act in, see MDP foundations. For direct policy optimization, see policy gradient. For the values they read, see value functions.

You need probability basics before this page. Probabilities are never negative and sum to 1. A distribution that reads [0.29, 0.08, 0.64] means the third move fires about 64 times in 100.

Think of It Like This

Playbook with mix ratios

A quarterback walks to the line and reads the defense. That look is the state. The playbook is the policy.

A rigid call says Cover 0 means quick slant, every time. That is deterministic: one state, one move. A mixed call says Cover 2 means 80 percent handoff, 15 percent short pass, 5 percent deep shot. That is stochastic: one state, a mix. The mix stops the other sideline from calling your play before you run it.

Where the analogy stops: the quarterback got the book from coaches. Your agent starts with random weights and writes its own book from rewards, one backup at a time.

How It Actually Works

Write s for state, a for action, mu for a deterministic policy, pi for a stochastic one. A deterministic policy maps states to moves as a equals mu(s). A stochastic policy maps states to chances as pi(a given s), with each entry at or above 0 and entries summing to 1. In continuous control the same idea becomes a density, often a Gaussian whose mean and spread come from networks.

From values to choices

Value methods learn Q(s, a) then derive behavior three ways. Greedy picks argmax Q, the top value. Epsilon greedy picks greedy with chance 1 minus epsilon and a random move otherwise, which forces a little exploration. Softmax turns values into chances with temperature tau:

$\\pi(a \\mid s) = \\frac{\\exp(Q(s, a) / \\tau)}{\\sum_b \\exp(Q(s, b) / \\tau)}$

High tau flattens toward uniform. Low tau sharpens toward greedy. Tau must stay above 0.

Optimal policy pi star means no other policy earns more from any state, written V pi star (s) above or equal V pi (s) for all s and pi. Fully observed MDPs admit a deterministic optimum, yet training still needs stochasticity to find it, and games or partial views often need it forever.

Worked example

Yellow light, three moves, learned values Q of 2.40 for hold speed, 1.10 for brake, 3.20 for go. Greedy picks go because 3.20 tops the list.

At tau 1.0, exponents are exp(2.40) 11.0232, exp(1.10) 3.0042, exp(3.20) 24.5325. The normalizer Z is 38.5599. Chances are 11.0232 divided by 38.5599 which is 0.2859, 3.0042 divided by 38.5599 which is 0.0779, and 24.5325 divided by 38.5599 which is 0.6362. They sum to 1.0000.

Cool the temperature to 0.3 and go dominates near 0.93. Heat it to 4.0 and the mix flattens toward thirds, with go near 0.44, hold near 0.32, brake near 0.23. Same values, wider search. The diagram draws these three bars at tau 1.0 with the greedy arrow on go.

Code

This snippet recomputes the mix and checks the sum.

1import numpy as np2
3
4def softmax(q: np.ndarray, tau: float = 1.0) -> np.ndarray:5    """Boltzmann chances with max subtraction for stability."""6    if tau <= 0:7        raise ValueError("tau must stay positive")8    z = (q - np.max(q)) / tau9    e = np.exp(z)10    return e / np.sum(e)11
12
13q = np.array([2.40, 1.10, 3.20])14p = softmax(q, tau=1.0)15print([f"{v:.4f}" for v in p])16# -> ['0.2859', '0.0779', '0.6362']17print(f"{float(np.sum(p)):.4f} {int(np.argmax(q))}")18# -> 1.0000 2

Watch Out For

Deterministic play where mixing is required

The symptom is an agent that looks solved in training and gets picked apart on release. In poker, rock paper scissors, or a symmetric corridor with thin sensors, a fixed move is readable. The opponent or the aliasing exploits it every time.

Fix it by keeping a stochastic head where opponents or hidden state exist, or by adding memory such as a recurrent net so identical looks can still produce different moves. Deterministic is for deployment in fully seen worlds, not for every world.

Entropy collapse that ends search early

The symptom is a lucky early win that becomes the only move. Logits for one action spike, its chance nears 1.0, rivals stop getting sampled, and their gradients starve. The agent locks a local best.

Fix it with an entropy bonus beta times H(pi) added to the objective, where H is minus sum pi log pi. PPO and Soft Actor Critic both use this form. Watch entropy during training: if it dives while returns plateau, raise the bonus before touching the learning rate.

The Quick Version

A policy turns states into actions, as one fixed move or as chances that sum to 1.
Greedy takes the top Q, epsilon greedy mixes in random moves, softmax mixes by temperature tau.
The worked values give chances 0.2859, 0.0779, and 0.6362 at tau 1.0, with greedy on the third move.
Fully seen MDPs admit a deterministic optimum, but search and hidden state still need stochasticity.
Readable fixed play loses in games, and early entropy collapse locks in a local best unless you bonus entropy.