Optimistic Initial Values

Start every action value absurdly high, and a purely greedy agent will still tour each option once, because every real reward feels like a letdown that sinks the tried below the untried.

Starting every arm at 5.0 forces a greedy agent to sample each arm once, since each reward drags its arm below the untried ones

Why Does This Exist?

A purely greedy agent picks whatever currently ranks highest, and with all values starting at zero it locks on fast. The first arm to pay anything positive jumps above the untried zeros, so the agent replays it forever and never learns the neighbor pays four times more. That lock-in is the whole exploration problem in miniature.

Random exploration fixes it with a permanent tax: epsilon-greedy keeps pulling losing arms at rate $\varepsilon$ for the rest of time. Optimistic initialization buys the early exploration without the tax. Set every starting estimate above anything the environment can pay, $Q_1(a) = Q_0 > R_{\max }$ , and each pull disappoints by construction. The tried arm sinks below the pristine untried ones, so pure greed tours the whole set before repeating anything.

This page is the bandit-scale version. For the problem class it lives in, see multi-armed bandits. For the randomized alternative, see epsilon-greedy. For optimism that never burns out, see upper confidence bound.

The one idea you need first

A constant step-size update $Q \leftarrow Q + \alpha(R - Q)$ moves the estimate a fraction $\alpha$ toward each observed reward. Unroll it and the starting value decays geometrically: after $n$ pulls of one arm its weight is $(1 - \alpha)^n$ , so the initial bias washes out and $Q$ settles near the arm's true mean. Optimism is a prior with an expiry date, which is both its charm and its limit.

Think of It Like This

The critic who assumes five stars everywhere

A food critic moves to a new city convinced every restaurant is a five-star masterpiece until proven otherwise. Monday she tries place A expecting 5.0 and gets a 3.0 meal. Disappointed, she marks A down to 4.2. Tuesday she asks where to eat: A sits at 4.2 while B, C, and D still hold their untouched 5.0, so pure greed walks past A into B. B is a 1.0 dive, which craters its score. Wednesday and Thursday go to C and D for the same reason.

No coin flips, no dithering. The optimism guarantees each early meal underwhelms, and the disappointment steering tours every kitchen in town before anything gets a second visit. The best kitchen wins in the end because it disappoints the least.

Where it stops: kitchens change chefs. A year later B hires a star and becomes the best table in town, but the critic's score for B is parked near 1.0 and she never walks in. Optimism burns once; it does not relight when the world shifts underneath it.

How It Actually Works

With $Q_1(a) = Q_0 > R_{\max}$ for every arm and greedy selection $A_t = \arg\max_a Q_t(a)$ , the first prediction error on any arm is negative by construction: $R_t - Q_t(A_t) < 0$ . The update $Q \leftarrow Q + \alpha(R - Q)$ pushes the pulled arm down while untried arms sit untouched at $Q_0$ , so the next greedy pick must be an untried arm. After $k$ arms each get pulled once, the ranking reflects real evidence: the arm that disappointed least ranks highest.

The bias decays as $(1 - \alpha)^n$ with $n$ pulls, so estimates converge to true means and the touring stops on its own. No schedule, no coin, no tuning beyond choosing $Q_0$ somewhat above the best plausible reward.

Worked example

Three arms with true means 0.80, 0.40, 0.10. Start all at $Q_0 = 5.0$ , step size $\alpha = 0.1$ , ties broken to the lowest index:

Step 1: all tied at 5.0, pick arm 1, reward 0.80. $Q = 5.0 + 0.1(0.80 - 5.0) = 5.0 - 0.42 = 4.58$ . Table: $[4.58, 5.00, 5.00]$ .
Step 2: arms 2 and 3 tied above arm 1, pick arm 2, reward 0.40. $Q = 5.0 + 0.1(0.40 - 5.0) = 5.0 - 0.46 = 4.54$ . Table: $[4.58, 4.54, 5.00]$ .
Step 3: arm 3 alone on top, pick arm 3, reward 0.10. $Q = 5.0 + 0.1(0.10 - 5.0) = 5.0 - 0.49 = 4.51$ . Table: $[4.58, 4.54, 4.51]$ .
Step 4: arm 1 leads because it disappointed least. Reward 0.80 again: $Q = 4.58 + 0.1(0.80 - 4.58) = 4.58 - 0.378 = 4.202$ . Table: $[4.202, 4.54, 4.51]$ , so step 5 turns back to arm 2.

Every arm got sampled once with zero randomness, and the best arm earned its lead honestly.

Code

This replay uses the exact fixture above and prints the picks plus the table:

1def greedy_run(true_means: list[float], q0: float, alpha: float, steps: int) -> tuple[list[int], list[float]]:2    """Pure greedy bandit with constant step size; ties go to lowest index."""3    q = [q0] * len(true_means)4    picks: list[int] = []5    for _ in range(steps):6        best = 07        for arm in range(1, len(q)):8            if q[arm] > q[best]:9                best = arm10        picks.append(best)11        q[best] = q[best] + alpha * (true_means[best] - q[best])12    return picks, [round(v, 3) for v in q]13
14
15print(greedy_run([0.8, 0.4, 0.1], 5.0, 0.1, 4))  # -> ([0, 1, 2, 0], [4.202, 4.54, 4.51])

The picks tour all three arms before repeating, and the final table matches the hand computation.

Watch Out For

Optimism burns out in shifting environments

The touring works exactly once. After $(1 - \alpha)^n$ decays, every arm's estimate sits near its sampled mean and greed stops exploring. If payouts drift later, say a weak arm becomes the best one in week two, the agent never notices: that arm's score is parked low and nothing pulls it back up for a retest.

Never use optimism as the only exploration in a nonstationary problem. Pair it with a persistent floor like epsilon-greedy, a sliding window, or a UCB bonus that keeps questioning old scores.

A cartoonish Q0 wastes hundreds of pulls

If $Q_0$ towers over reality, say 1000 against rewards near 1, the $(1 - \alpha)^n$ decay takes forever and the agent keeps sampling plainly bad arms because their scores are still inflated. The touring degenerates into a long parade of known losers.

Set $Q_0$ modestly above the plausible ceiling, around 1.5 to 2 times the best reward you expect. The disappointment steering only needs untried arms to outrank tried ones, not to orbit the moon.

The Quick Version

Optimistic initialization sets every $Q_1(a)$ above any achievable reward, so each pull disappoints and sinks below the untried arms.
A purely greedy policy then tours every arm once with zero randomness; here 5.0 drops to 4.58, 4.54, and 4.51 across three picks.
The starting bias decays as $(1 - \alpha)^n$ , so estimates converge to true means and the touring ends by itself.
Optimism fires once only, so shifting environments need a persistent explorer alongside it.
Keep $Q_0$ modestly above the reward ceiling; a giant $Q_0$ wastes pulls on obvious losers.