The Options Framework

One call can run a whole skill instead of one muscle twitch. An option says where it can start, what it does while it runs, and when it hands control back.

A doorway option runs two steps for return 0.90, discounts the future by 0.81, and sets an SMDP target of 5.76

Why Does This Exist?

Flat agents pick one primitive action per tick. On a long task, say crossing four rooms to a doorway, that means thousands of tiny decisions before any reward arrives. Random exploration almost never stumbles through the whole chain, and one delayed reward must crawl backward across every single step before early states learn anything.

The Options Framework answers with temporal abstraction. It packages a whole skill, like go to the doorway, into one callable unit. The high-level policy then decides among skills instead of twitches, and each skill runs for as many steps as it needs. This is the foundation hierarchical RL builds on: this page defines the option itself, while the sequel that learns option boundaries end to end is the option-critic architecture.

Think of It Like This

The airport shuttle with fixed stops

You stand outside arrivals and a shuttle pulls up with a sign reading Downtown Express. That sign tells you three things. It boards only at marked stops, which is the initiation set. While aboard, the driver follows one fixed route with no input from you, which is the internal policy. And it lets you off only at its terminus, where you pick your next ride, which is the termination condition.

You still travel every meter of road, just as the agent still executes every primitive step. But your planning happens at the level of rides, not footsteps: shuttle, then train, then walk. Two rides replace forty micro-decisions.

Where the analogy stops: shuttle routes are fixed and safe. An option runs in a stochastic world, its termination chance is evaluated fresh in every state, and a badly drawn skill can skip the best path entirely.

How It Actually Works

An option $\omega$ is a triple:

\omega = \langle \mathcal{I}_{\omega}, \pi_{\omega}, \beta_{\omega} \rangle

In plain words: $\mathcal{I}_{\omega}$ is the set of states where the option may start. $\pi_{\omega}(a \mid s)$ is the internal policy, the chance of taking primitive action $a$ in state $s$ while the option runs. $\beta_{\omega}(s)$ is the termination chance evaluated on arrival in each state $s$ : with probability $\beta_{\omega}(s)$ the option ends and the high-level policy picks again, otherwise it keeps running. A primitive action is just a one-step option with $\mathcal{I} = \mathcal{S}$ and $\beta = 1$ everywhere, so nothing from the flat setup is lost.

Because options last a variable $k$ steps, the macro level is a semi-Markov decision process. The reward piles up with discounting across the run:

R(s, \omega) = \mathbb{E}\left[ r_{t+1} + \gamma r_{t+2} + \dots + \gamma^{k-1} r_{t+k} \right]

And the macro Bellman equation discounts the future by $\gamma^k$ instead of $\gamma$ :

Q(s, \omega) = R(s, \omega) + \gamma^k \max_{\omega'} Q(s', \omega')

Intra-option learning is the efficiency trick. Instead of waiting $k$ steps for the option to end, it updates on every primitive transition with a target that blends continuing against terminating:

y = r + \gamma \left[ (1 - \beta_{\omega}(s')) Q(s', \omega) + \beta_{\omega}(s') \max_{\omega'} Q(s', \omega') \right]

If $\beta = 0$ the target keeps the same option's value; if $\beta = 1$ it switches to the best next option.

Worked example

A doorway skill $\omega$ starts in room 1 with discount $\gamma = 0.90$ and current estimate $Q(s_0, \omega) = 5.00$ . It runs $k = 2$ steps: step 1 moves east for reward $r_1 = 0.0$ and continues ( $\beta = 0.0$ ), step 2 moves east for $r_2 = 1.0$ and terminates at the doorway ( $\beta = 1.0$ ). The best next option there is worth $6.00$ .

Return over the run:

R = 0.0 + 0.90 \times 1.0 = 0.90

Effective discount $\gamma^2 = 0.81$ . SMDP target:

y = 0.90 + 0.81 \times 6.00 = 0.90 + 4.86 = 5.76

Against the old estimate $5.00$ the TD error is $+0.76$ and the half-squared loss is $0.5 \times 0.76^2 = 0.2888$ . The intra-option target on the final step agrees from the one-step view: $1.0 + 0.90 \times 6.00 = 6.40$ , learned immediately without waiting to stitch the trajectory.

Code

A small check that recomputes the example. No libraries beyond the standard one.

1import math2
3gamma = 0.904rewards = [0.0, 1.0]5
6r_smdp = sum((gamma**t) * r for t, r in enumerate(rewards))7eff_gamma = gamma ** len(rewards)8target = r_smdp + eff_gamma * 6.009td_error = target - 5.0010loss = 0.5 * td_error**211intra_step2 = 1.0 + gamma * 6.0012
13print(f"R = {r_smdp:.2f}")14print(f"gamma^k = {eff_gamma:.2f}")15print(f"target = {target:.2f}")16print(f"td error = {td_error:+.2f}")17print(f"loss = {loss:.4f}")18print(f"intra-option step 2 = {intra_step2:.2f}")19
20assert math.isclose(r_smdp, 0.90)21assert math.isclose(eff_gamma, 0.81)22assert math.isclose(target, 5.76)23assert math.isclose(td_error, 0.76)24assert math.isclose(loss, 0.2888)25assert math.isclose(intra_step2, 6.40)26print("checks passed")

1R = 0.902gamma^k = 0.813target = 5.764td error = +0.765loss = 0.28886intra-option step 2 = 6.407checks passed

Watch Out For

Options that switch every step

The symptom is hierarchy with no benefit: the high-level policy changes options at nearly every tick, so $k = 1$ everywhere and the SMDP collapses back into a flat MDP plus overhead. The cause is a termination chance set too high, so every option quits before it does anything useful.

Fix it by pushing termination down and charging a small switching cost for each new option call. Long, committed runs should be the default; switching should have to earn its keep.

Options that never hand back control

The mirror failure is the zombie skill: termination stays near zero, the option loops in one corner of the building, and the high-level policy never gets to act. Returns flatline while one option burns the whole budget.

Fix it by testing termination separately from the policy. Roll each option out alone with a step cap and confirm it ends at its stated goal from every start in its initiation set. If it cannot finish alone, no planner above it can save the run.

The Quick Version

An option is the triple initiation set, internal policy, termination chance. It starts where allowed, steers each step, and hands back control when its chance fires.
Options turn the macro level into a semi-Markov process, so multi-step rewards accumulate with discounting and the future discounts by gamma to the power of the run length.
The doorway run earns return 0.90, discounts by 0.81, and sets target 5.76 against an old estimate of 5.00 for an error of plus 0.76.
Intra-option updates learn from every primitive step instead of waiting for termination, which is what makes long skills affordable to train.
Termination that fires too often destroys the abstraction, and termination that never fires locks out the planner, so test each option solo before trusting the hierarchy.