Feudal Networks (FuN)

A slow Manager points where the situation should drift over the next few steps, and a fast Worker earns pay for moving that way. Direction only, no micromanagement.

A Manager points north for two steps while the Worker moves through [1.0, 0.6] to [1.2, 1.6], earning cosine pay 0.50 and 0.50 before the Manager banks advantage plus 2.00

Why Does This Exist?

Some tasks pay only at the end of very long sequences: thousands of small moves, one score. Flat agents fail there because random motor noise almost never stumbles through the full sequence, so the chance of seeing even one rewarding trajectory shrinks toward zero as the horizon grows. Credit assignment across thousands of steps is the wall.

Hierarchy is the natural answer: split control into slow planning and fast acting. But the early versions each carried a tax. Handbuilt hierarchies needed engineers to name every subgoal by hand. Option-style methods had to learn when each skill starts and stops, and the skills often collapsed into single actions. The original feudal formulation hid sensory detail from the upper level, throttling communication between the layers.

Feudal Networks, introduced by Vezhnevets and colleagues at DeepMind in 2017, keep the hierarchy and drop the taxes. The Manager issues directions as unit vectors in a learned latent space every $c$ steps, typically $c = 10$ . The Worker chases those directions for cosine-shaped intrinsic pay and never sees the outside reward. And the Manager learns from a transition policy gradient that needs no backpropagation through the Worker's actions.

This page covers the FuN mechanism end to end. The general hierarchy story is hierarchical reinforcement learning, discrete skills with initiation and termination are the options framework and the option-critic, and value decomposition across a hierarchy is MAXQ, in the read-next list.

The one idea you need first.

Hierarchical RL splits a policy into at least two levels: an upper level that decides what should happen over an extended stretch, and a lower level that converts each decision into primitive actions. The upper level runs at a coarser clock, so one of its decisions spans many environment steps. FuN is one concrete way to wire the two levels together and train them both.

Think of It Like This

A film director and a camera operator

A director watches the monitor and calls the next move: dolly right, hold two seconds. The operator handles focus, footing, and framing around the furniture, things the director never touches. The operator is paid for how faithfully the shot follows the call, not for box office numbers. At the premiere, if the film lands, the director learns to call more moves like that one.

Point for point: the call is the goal vector $g_t$ , the shot drifting across the frame is the latent displacement, the operator's paycheck is the cosine intrinsic reward, and the box office is the extrinsic return that trains only the director. Where it stops: a real crew shares one monitor and can renegotiate in words mid-shot, while FuN fixes the protocol to vectors and cosines. And a bad take wastes an afternoon, whereas a bad Worker rollout in a live system can break hardware, which is why intrinsic pay is tuned, never blindly maximized.

How It Actually Works

A slow Manager and a fast Worker

Raw observations $s_t$ pass through a shared perception network into a compact latent state $z_t$ . Here $z_t$ is the agent's internal summary of the situation, $g_t$ is the Manager's goal vector, $c$ is the number of primitive steps per Manager tick, $a_t$ is a primitive action, $R_t$ is outside reward, and $r^I_t$ is the Worker's intrinsic pay.

The Manager wakes every $c$ steps and emits one goal $g_t$ , normalized to unit length $\hat{g}_t = g_t / \|g_t\|$ . Crucially the goal is a direction, not a destination: it orders the Worker to shift the latent state that way over the coming stretch, without naming any coordinates to arrive at. The Worker acts every step from a policy $\pi(a_t \mid z_t, w_t)$ , where $w_t$ is an embedding of the current goal, and handles reflexes, obstacles, and actuation.

Both levels read the same $z_t$ . Nothing is hidden from either side, so abstraction comes from the different clocks and jobs, not from concealing information.

The Worker's pay is cosine alignment

The Worker never sees $R_t$ . Its pay at step $t$ averages how well recent motion lined up with recent orders:

$r^I_t = \frac{1}{c} \sum_{i=1}^c \cos\left(z_t - z_{t-i},\, g_{t-i}\right)$

with $\cos(u, v) = (u \cdot v) / (\|u\|\|v\|)$ . Motion exactly along the order scores $+1$ , sideways scores $0$ , and directly against scores $-1$ , all scaled by $1/c$ . The Worker maximizes this dense signal with any standard policy gradient method, so it gets step-by-step guidance even where outside reward is silent for thousands of steps.

The Manager learns from transitions, not backprop

Training the Manager by differentiating through $c$ Worker actions would be slow and fragile. Instead FuN treats the Worker's compliance as a transition model: goals that the Worker obeys make matching displacements likely, $p(z_{t+c} \mid z_t, g_t) \propto \exp(\cos(z_{t+c} - z_t, g_t))$ . The policy gradient theorem applied to that distribution gives the transition policy gradient:

$\nabla_{g_t} J_M = \mathbb{E}\left[ \left(R_t - V_M(s_t)\right) \nabla_{g_t} \cos\left(z_{t+c} - z_t,\, g_t\right) \right]$

where $R_t$ is the discounted extrinsic return from $t$ onward, $V_M(s_t)$ is the Manager's baseline value, and their difference $A_M = R_t - V_M(s_t)$ is the Manager's advantage. In plain words: when a stretch paid off ( $A_M > 0$ ), rotate the goal toward the displacement that actually happened. Goals that preceded winning motion get reinforced directly.

Worked example

Horizon $c = 2$ , latent dimension $2$ . Start $z_0 = [1.0, 0.0]$ , Manager goal $g_0 = [0.0, 1.0]$ , straight north.

Step 1: the Worker reaches $z_1 = [1.0, 0.6]$ . Displacement $[0.0, 0.6]$ against goal $[0.0, 1.0]$ : dot product $0.60$ , norms $0.60$ and $1.00$ , cosine $1.0000$ . Pay $r^I_1 = 1.0000 / 2 = 0.5000$ .

Step 2: the Worker reaches $z_2 = [1.2, 1.6]$ . Displacement from the macro origin is $[0.2, 1.6]$ : dot with the goal is $1.60$ , norm $\sqrt{0.04 + 2.56} = \sqrt{2.60} \approx 1.61245$ , cosine $1.60 / 1.61245 \approx 0.9923$ . Pay $r^I_2 = 0.9923 / 2 \approx 0.4961$ .

Manager update: outside return $R_2 = 5.00$ against baseline $V_M = 3.00$ gives advantage $A_M = +2.00$ . Positive, so the gradient rotates $g_0$ toward the realized $[0.2, 1.6]$ displacement. The Manager learns that pointing north-ish here precedes payoff.

Code

The goal math, the Worker's pay, and the Manager's advantage, using only the standard library. Display values are rounded; checks use tolerance.

1import math2from typing import List3
4
5def cosine(a: List[float], b: List[float]) -> float:6    """Cosine similarity between two vectors."""7    dot = sum(x * y for x, y in zip(a, b))8    na = math.sqrt(sum(x * x for x in a))9    nb = math.sqrt(sum(y * y for y in b))10    return dot / (na * nb) if na and nb else 0.011
12
13z0 = [1.0, 0.0]14g0 = [0.0, 1.0]15c = 216z1 = [1.0, 0.6]17z2 = [1.2, 1.6]18d1 = [b - a for a, b in zip(z0, z1)]19d2 = [b - a for a, b in zip(z0, z2)]20cos1, cos2 = cosine(d1, g0), cosine(d2, g0)21print([round(v, 2) for v in d2])  # -> [0.2, 1.6]22print(f"{cos1:.4f} {cos1 / c:.4f}")  # -> 1.0000 0.500023print(f"{cos2:.4f} {cos2 / c:.4f}")  # -> 0.9923 0.496124advantage = 5.00 - 3.0025print(f"{advantage:+.2f}")  # -> +2.0026assert math.isclose(cos1, 1.0) and math.isclose(cos2, 0.99227787, rel_tol=1e-6)27assert math.isclose(advantage, 2.0)

Every printed number matches the hand computation: perfect alignment first, $0.9923$ then, half pay each step, advantage $+2.00$ .

Watch Out For

Letting policy gradients reshape the goal space mid-training

Goals live in the perception space $z_t$ , so if Manager or Worker gradients also train the perception network, the meaning of every goal vector drifts. A command that meant one motion on Monday means another by Friday, and previously skilled Worker behavior misaligns for no visible reason.

The symptom is intrinsic pay swinging wildly after early mastery, with sudden performance cliffs unrelated to any reward change. The fix is a stop-gradient: perception learns only from auxiliary self-supervised objectives such as next-frame prediction, never from the policy losses. Stable geometry first, control second.

A Worker that stops listening to the goal

If the intrinsic weight is too small next to other losses, or the goal embedding is badly scaled, the Worker optimizes everything except goal alignment: cosine hovers near zero regardless of orders, and behavior with the goal zeroed out looks identical.

The symptom is a flat cosine histogram plus a Manager whose goals wander without effect. The fix is to check the cosine variance before touching rewards: scale the intrinsic term until alignment clearly moves behavior, confirm the goal embedding has nonzero gradients, and only then tune the Manager's step size. A hierarchy where the lower level ignores orders is just two networks burning compute.

The Quick Version

FuN splits control into a Manager ticking every $c$ steps and a Worker acting every step, both reading the same latent state with nothing hidden.
Manager goals are unit direction vectors: shift the situation this way, with no destination coordinates attached.
The Worker earns only cosine alignment between its motion and its orders, giving dense guidance where outside reward is silent.
The Manager trains by transition policy gradient: positive advantage rotates the goal toward the displacement that actually paid off.
In the worked macro-step, alignment $1.0000$ then $0.9923$ pays $0.5000$ and $0.4961$ , and advantage $+2.00$ reinforces the north-ish order.
Block policy gradients from reshaping perception, and verify the Worker actually listens before tuning the Manager.