Skip to content
AI360Xpert
Beta
Core ML

Visual explainer

Mixture Of Experts

Eight experts share the work while each token wakes only two, so the model holds far more knowledge than it spends compute on.

One token wakes all eight experts and pays eight times over.
One token wakes all eight experts and pays eight times over.

A dense layer wakes every weight for every token, so doubling knowledge doubles the bill. Eight experts sit ready, but firing all of them for one word is pure waste you feel in the bill.

Gates Pick Two

The router scores four experts and wakes only the top two.
The router scores four experts and wakes only the top two.

Your router scores four experts at 0.50, 0.30, 0.12 and 0.08, then keeps only the top two. The rest sleep through this token entirely.

Blend by Weight

Gates 0.50 and 0.30 renormalize to blend outputs into 2.5 and 3.0.
Gates 0.50 and 0.30 renormalize to blend outputs into 2.5 and 3.0.

Gates 0.50 and 0.30 share 0.80 of mass, so they renormalize to 0.625 and 0.375. E1 says [4, 0], E2 says [0, 8], and the token leaves as [2.5, 3.0].

Even Load Matters

A balancing loss spreads 32 tokens evenly instead of starving three experts.
A balancing loss spreads 32 tokens evenly instead of starving three experts.

Thirty-two tokens split 8 each across four experts when the balancing loss does its job. Without it, one expert hogs 29 and the rest forget how to help on real traffic.

Quarter of the Bill

Two awake experts cost a quarter of eight awake experts.
Two awake experts cost a quarter of eight awake experts.

The same eight slots fire all at once in a dense layer and two at a time here. You keep the same knowledge and pay a quarter of the compute.

One or Two

Top-1 routing is cheapest until stalled quality forces top-2.
Top-1 routing is cheapest until stalled quality forces top-2.

Top-1 routing costs half of top-2 and works until quality stalls. When your loss flatlines and one expert looks unsure on hard tokens, pay for the second opinion.

Where It Breaks

One lucky expert grabs every token and the other seven starve.
One lucky expert grabs every token and the other seven starve.

Early luck can crown one expert, which then grabs every token and every gradient. The rest starve untrained, and your eight experts quietly become one.

The Quick Version

  • Dense layers wake everything for every token.
  • The router keeps only the top two experts.
  • Gates renormalize to 0.625 and 0.375.
  • Balanced load gives each expert 8 tokens.
  • Two awake cost a quarter of eight.
  • Default top-1, upgrade when quality stalls.
  • Collapse turns eight experts into one.