Policy Objective Functions

An objective function says what a good policy means: total return from the start, average value over visits, or reward per step forever.

Three objectives score the same switching policy at 1.00, 1.33 and 1.50 reward per step, and all three peak at always switch

Why Does This Exist?

Policy gradients climb $J(\theta)$ , but what is $J$ ? Value methods hide their target inside Bellman error, while policy methods must name the number they maximize. The honest answer depends on the task shape: a game ends, a server queue never does, and a refinery cares about throughput per hour, not discounted sums.

Pick the wrong score and the agent optimizes the wrong thing. Discounting a never ending task invents a horizon that is not there; scoring a game by steady state throughput ignores that only the final result counts. This page defines the three standard scores, shows they share one gradient form, and works one policy under all three so you can feel the difference.

Think of It Like This

Three ways to judge a night market stall

A stall owner can be judged three ways. Total profit from opening night to closing day suits a one week festival stall. Average till balance across evenings, weighted by how often each crowd shows up, suits a seasonal stall that discounts later weeks. Profit per hour suits a permanent stall that never closes.

Same stall, same cooking, three scoreboards. The permanent stall owner who chases opening night totals will overstaff week one and burn out. Match the scoreboard to the stall's lifespan, or the numbers will lie to you.

The analogy stops where math starts: market crowds shift with seasons, while these objectives assume the environment's dynamics stay fixed.

How It Actually Works

Write the parameterized policy $\pi_\theta(a \mid s)$ and let $d^{\pi}(s)$ be how often the policy visits $s$ . Three scores cover the task shapes:

Start value $J_1(\theta) = V^{\pi_\theta}(s_0)$ for episodic tasks: expected discounted return from the start state $s_0$ . Visits are counted with discounting, $d(s) = \sum_t \gamma^t P(S_t = s \mid s_0)$ .
Average value $J_{avV}(\theta) = \sum_s d^{\pi}(s) V^{\pi}(s)$ for continuing discounted tasks: state values weighted by the stationary visit frequencies.
Average reward $J_{avR}(\theta) = r(\pi) = \sum_s d^{\pi}(s) \sum_a \pi(a \mid s) \mathcal{R}_s^a$ for continuing undiscounted tasks: plain reward per step, where $\mathcal{R}_s^a$ is the expected immediate reward.

Differentiating looks hopeless because $\theta$ moves both the actions and the visit frequencies. The policy gradient theorem removes the second dependence: for all three scores,

\nabla_\theta J(\theta) \propto \sum_s d^{\pi}(s) \sum_a \nabla_\theta \pi_\theta(a \mid s) \, Q^{\pi}(s, a)

which rewrites with the log trick as an expectation over sampled steps, $\mathbb{E}[\nabla_\theta \log \pi_\theta(A_t \mid S_t) \, Q^{\pi}(S_t, A_t)]$ . That single form is what REINFORCE and actor critic methods sample.

Worked example

Two market nights cycle: night 1 offers safe stir fry (stay, reward 1) or fancy seafood (switch to night 2, reward 3); night 2 always returns with reward 0. Let $p$ be the switch probability, discount $\gamma = 0.8$ . Visit frequencies solve to $d_1 = 1 / (1 + p)$ and $d_2 = p / (1 + p)$ .

At $p = 0.0$ (never switch): visits $[1.000, 0.000]$ , start value $5.000$ , average value $5.000$ , reward rate $1.000$ .
At $p = 0.5$ (mix): visits $[0.667, 0.333]$ , start value $7.143$ , average value $6.667$ , reward rate $1.333$ .
At $p = 1.0$ (always switch): visits $[0.500, 0.500]$ , start value $8.333$ , average value $7.500$ , reward rate $1.500$ .

Check one by hand: at $p = 0.5$ the reward rate is $(1 + 2 \times 0.5) / 1.5 = 2.0 / 1.5 = 1.333$ . All three scoreboards rise together and crown $p^* = 1.0$ . They disagree on units but agree on the winner, which is exactly why one gradient serves all three.

Code

This snippet evaluates all three scores for any switch probability using the closed forms above.

1def objectives(p: float, gamma: float = 0.8):2    """Closed form scores for the two night market: never, mix, always switch."""3    d1 = 1.0 / (1.0 + p)4    d2 = p / (1.0 + p)5    reward_rate = (1.0 + 2.0 * p) / (1.0 + p)6    start_value = (1.0 + 2.0 * p) / ((1.0 - gamma) + p * gamma * (1.0 - gamma))7    avg_value = d1 * start_value + d2 * gamma * start_value8    return {"d": [round(d1, 3), round(d2, 3)], "J1": round(start_value, 3),9            "JavV": round(avg_value, 3), "JavR": round(reward_rate, 3)}10
11
12print(objectives(0.0))13print(objectives(0.5))14print(objectives(1.0))

Watch Out For

Discounting a task that never ends

Applying discounted scores to a permanent process invents a fake horizon: early rewards count, late ones fade, and the agent chases startup transients instead of steady throughput.

Symptom: the learned policy depends on the arbitrary start state and the arbitrary discount. Fix: switch to the average reward score with differential returns, $R - r(\pi)$ , which measure each reward against the steady rate instead of against zero.

Averaging values under the wrong visits

Average value weights states by visit frequency, so a stale frequency estimate silently reweights the whole objective toward places the new policy never goes.

Symptom: the score improves on paper while the deployed policy earns less. Fix: re-estimate visit frequencies from fresh rollouts after every substantial policy change, and distrust averages computed under last month's behavior.

The Quick Version

Start value suits episodes, average value suits discounted continuing tasks, reward rate suits permanent ones.
All three gradients share one form, so one sampler optimizes whichever scoreboard fits.
Here never, mix, and always switch scored reward rates 1.000, 1.333, and 1.500.
All three scoreboards peaked at always switch, disagreeing on units but agreeing on the winner.
Discounting a never ending task fakes a horizon; use reward rate with differential returns instead.