Skip to content
AI360Xpert
Core ML

Neural Architecture Search

Instead of hand-designing a network, define a space of possible layer choices and let a search strategy try candidates, scored cheaply, to find a good one.

A search strategy repeatedly proposes candidate architectures from the space of possible layer choices, a fast proxy scores each one, and the best-scoring candidate is what finally gets trained in full
A search strategy repeatedly proposes candidate architectures from the space of possible layer choices, a fast proxy scores each one, and the best-scoring candidate is what finally gets trained in full

Why Does This Exist?

A team building an on-device vision model has already tried the well-known hand-designed architectures — a few variants of ResNet, a couple of MobileNet configurations — and none hits the accuracy-versus-latency target the product needs. The next step, historically, is a researcher trying variations by hand: swap a 3x3 convolution for a 5x5 here, add a skip connection there, retrain, check, repeat. Each iteration takes a full training run to evaluate, and the space of reasonable variations is enormous.

CNN architectures shows how each major design — AlexNet, VGG, ResNet, EfficientNet — fixed one specific problem with its predecessor. Every one of those fixes was a person's insight. Neural architecture search asks whether a search procedure can find comparable or better fixes automatically, without a person proposing each candidate by hand.

Think of It Like This

A recipe-testing kitchen instead of one chef guessing

A chef developing a new dish tries variations by intuition: a bit more acid, a different herb, more or less salt, tasting after each change. It works, but it's slow, and it can only try one adjustment at a time before their palate needs a break.

A test kitchen automates the exploration instead: define the space of things that could vary — three acid options, five herb choices, a salt range — and have a system try many combinations, using a quick taste-test proxy to screen out the bad ones before committing a full multi-hour braise to any single combination. Only the promising few get the full cooking time. Neural architecture search is that test kitchen, applied to network layers instead of ingredients.

How It Actually Works

The three pieces: space, strategy, and cost

Every NAS method is defined by three choices. The search space is what's allowed to vary — which layer types are candidates at each position, how many layers, what connections between them. The search strategy is how candidates get proposed from that space — random search, evolutionary algorithms, reinforcement learning, or gradient-based methods. The cost model is how a candidate gets scored — and this is where naive NAS becomes impractical, because fully training even one candidate to convergence can take many GPU-hours, and a nontrivial search space contains thousands of candidates.

Why the search space explodes fast

A search space with 5 layer positions, each independently choosing among 4 op types (say, a 3x3 conv, a 5x5 conv, a skip connection, or a pooling op), already contains 45=1,0244^5 = 1{,}024 distinct architectures. Real search spaces are considerably larger than this toy example, with more layers and more choices per layer, and training every candidate from scratch to evaluate it is where naive NAS becomes computationally impossible.

Weight sharing: training one supernet instead of many candidates

The breakthrough that made NAS practical for ordinary compute budgets is weight sharing, sometimes called one-shot NAS. Instead of training each candidate architecture separately, build one large supernet containing every candidate operation at every layer position simultaneously, and train it once. A candidate architecture is then a specific path through the supernet — one operation chosen per layer — and its performance can be estimated using the shared weights already trained inside the supernet, without any additional training for that specific candidate.

This turns the search cost from "proportional to the number of candidates" into "proportional to training one network, regardless of how many candidates the space contains" — a difference that matters enormously as the search space grows.

DARTS (Differentiable Architecture Search) pushed weight sharing further by making the choice of operation at each layer a continuous, learnable weighting over all candidate operations rather than a discrete pick. That turns architecture search itself into something solvable with ordinary gradient descent — the same optimizer already training the network's weights also learns which operations matter most at each position, and the discrete architecture is read off at the end by keeping whichever operation received the highest weight. This is a meaningfully different computational profile than the reinforcement-learning-based search strategies that came before it, which needed many discrete trials to get a comparable signal.

Show Me the Code

Search space size grows exponentially with layer count, and weight sharing avoids paying that cost per candidate.

layers = 5choices_per_layer = 4  # e.g. 3x3 conv, 5x5 conv, skip, poolsearch_space_size = choices_per_layer ** layersprint(f"search space: {choices_per_layer}^{layers} = {search_space_size:,} architectures")
epochs, gpu_hours_per_epoch = 50, 0.5naive_cost = search_space_size * epochs * gpu_hours_per_epoch  # train every candidateone_shot_cost = epochs * gpu_hours_per_epoch  # train one shared supernet, onceprint(f"train-every-candidate cost: {naive_cost:,.0f} GPU-hours")print(f"one-shot weight-sharing cost: {one_shot_cost:,.0f} GPU-hours")print(f"ratio: {naive_cost / one_shot_cost:,.0f}x fewer GPU-hours with weight sharing")# -> search space: 4^5 = 1,024 architectures# -> train-every-candidate cost: 25,600 GPU-hours# -> one-shot weight-sharing cost: 25 GPU-hours# -> ratio: 1,024x fewer GPU-hours with weight sharing

The naive approach's cost scales directly with the search space size — double the space, double the compute. Weight sharing decouples the two entirely: the supernet trains once regardless of how many candidate paths it contains, which is exactly why one-shot methods made NAS viable outside of organizations with enormous compute budgets.

Watch Out For

Trusting weight-sharing performance estimates as if they were real training runs

A candidate's estimated accuracy inside a shared supernet doesn't always match its accuracy when trained in isolation, because the shared weights were optimized for the average of many candidate paths, not any single one. This ranking mismatch is a known limitation of one-shot NAS — the search can favor architectures that look good under weight sharing but underperform once trained standalone. Always retrain and verify the final selected architecture from scratch before trusting the search's ranking.

Ignoring hardware cost signals in the search objective

A search that optimizes purely for accuracy will happily find architectures with excellent accuracy and terrible latency or memory footprint on the target deployment hardware — nothing in a pure accuracy objective penalizes that. Include a hardware-aware cost term (latency, parameter count, or measured on-device timing) directly in the search objective if the deployment target has real constraints, rather than filtering after the fact and hoping something acceptable survives.

The Quick Version

  • NAS is defined by three choices: the search space of allowed operations, the search strategy that proposes candidates, and the cost model that scores them.
  • Naive NAS is computationally infeasible because the search space grows exponentially with layer count, and training each candidate from scratch is expensive.
  • Weight sharing trains one supernet containing every candidate operation, decoupling search cost from search space size.
  • DARTS makes architecture choice continuous and differentiable, letting ordinary gradient descent search the space alongside the network's own weights.
  • Weight-sharing performance estimates can rank architectures incorrectly relative to how they'd perform trained standalone — always verify the final pick with a full training run.
  • CNN Architectures is the lineage of hand-designed fixes that NAS is trying to automate the discovery of.
  • Model Pruning and Knowledge Distillation are the other major routes to a smaller, more efficient network, working on an already-designed architecture rather than searching for a new one.
  • Policy Gradient Methods covers the reinforcement-learning search strategy that early NAS work used before weight-sharing and gradient-based methods took over.
  • Quantization-Aware Training is a complementary efficiency lever — worth stacking with a searched architecture rather than a substitute for one.

Related concepts