Visual explainer

Hyperparameter Tuning

Spend sixteen runs wisely: random search tests sixteen learning rates where a grid tests four, then spendier methods take over.

One knob swings the loss by 0.40 while the other barely moves it 0.02.

You never know upfront which knob matters. Here the learning rate drags loss from 0.60 down to 0.20 while batch size nudges it 0.31 to 0.29. Tuning blind means betting runs on the wrong dial.

Grids Repeat

Sixteen grid runs test only four distinct learning rates.

A 4 by 4 grid burns sixteen trainings yet tries four learning rates, each repeated down its column. Twelve runs re-test values you already know. Alignment is the waste, not the count.

Random Covers

Sixteen random runs test sixteen distinct learning rates.

Drop the alignment and every run tries a fresh rate: sixteen trainings, sixteen distinct values. Whichever knob turns out to rule, random search has already carpeted it four times denser than the grid.

Learn Each Round

Four scored tries point the fifth at the valley near 0.29.

Random still throws blind. After scores 0.41, 0.38, 0.33, 0.29, a guiding model aims the fifth try at the valley while peeking far afield now and then. Each result sharpens the next guess instead of dying forgotten.

Cull The Weak

Eight starters shrink to one winner while total training costs 32, not 64.

Full trainings for all eight would cost 64 units. Instead give eight configs one epoch, keep four, double their budget, and repeat: 8 plus 8 plus 8 plus 8 equals 32. Losers reveal themselves cheaply, so stop paying them early.

Match Budget

Cheap runs go random and parallel, pricey runs earn a guiding model.

Pick by price tag. Cheap runs go random and parallel across machines. Hours-long runs earn a guiding model that picks each step. And when losers show early, cull them mid-race instead of funding full terms.

Where It Breaks

The best of 100 tries flatters its own scoreboard, so a sealed test decides.

The luckiest of 100 configs looks golden on the split it was picked from: 0.24 there, 0.31 on sealed data. Tuning is peeking, so the scoreboard lies upward. Tune on one split and judge on another you never touch.

The Quick Version

One knob usually dominates the rest.
Grids retest the same few values.
Random tries a fresh value per run.
Guidance aims each try from history.
Culling halves cost from 64 to 32.
Pricey runs earn smarter search.
Winners flatter their own scoreboard.