Deep Learning
3 interview questions in this topic. Expand any question to read the full answer.
“What is the vanishing gradient problem and how do you fix it?”
Answer
Backpropagation computes each layer's gradient by walking the chain rule backward from the loss. The gradient that reaches layer k is the product of every local derivative between the output and layer k:
∂L/∂θₖ ∝ ∏ (Wᵢ · σ'(zᵢ))
That product is the whole story. If the typical factor is less than 1 — which is exactly what saturating activations like sigmoid give you (its derivative peaks at just 0.25) — the product shrinks geometrically with depth, and a 30-layer network can hand its first layer a gradient that's effectively zero. Those early layers stop learning the low-level features everything else depends on. That's the vanishing gradient problem. The mirror image, factors consistently greater than 1, gives exploding gradients.
The symptoms are recognizable: training loss drops then flatlines early, the first layers' weights barely move while the last layers train fine, and adding more depth makes things worse.
The fixes all keep that product near 1:
- Non-saturating activations like ReLU, whose derivative is exactly 1 for positive inputs.
- Residual / skip connections (ResNets), which give the gradient an identity path straight through — the trick that made 100-plus-layer networks trainable.
- Careful initialization (He, Xavier) and normalization (batch/layer norm) to keep each layer's signal at a healthy scale.
- For RNNs, gated architectures (LSTM, GRU) carry an additive cell state so gradients flow through time. And gradient clipping caps the norm to tame the exploding-gradient twin.
💡 Note It's a game of telephone down a long line of people. Every person passes the message along a little more quietly, and by the time it reaches the front there's nothing left to hear. The vanishing gradient is that whisper fading to silence before it reaches the early layers.
Related Questions
“What is batch normalization and why does it help?”
Answer
Deep networks have a moving-target problem: as early layers update, the distribution of inputs that later layers see keeps shifting, so every layer chases a target that won't sit still. Batch normalization stabilizes that by fixing the scale of each layer's inputs. For each feature in a mini-batch it does two things:
x̂ = (x − μ_batch) / √(σ²_batch + ε), then y = γ · x̂ + β
First normalize to zero mean and unit variance over the batch (the ε just avoids dividing by zero), then scale and shift with two learned parameters, γ and β. That last step matters: forcing every layer to zero mean and unit variance would throw away useful information, so γ and β are the network's escape hatch — it can recover any mean and variance it wants, so normalization never costs expressive power.
Why it helps: activations stay at a stable scale, so you can train with higher learning rates and converge faster, the model is less sensitive to initialization, and because the batch statistics come from a random mini-batch, each example is normalized slightly differently every step — a mild regularization effect.
The one thing you must get right is train vs. inference. At training time you use the batch's own statistics; at test time batch norm switches to a running average accumulated during training. Forgetting to switch modes is a classic bug. A related note: layer normalization normalizes across the features of a single example, so it doesn't depend on batch size — which is why Transformers use layer norm, not batch norm.
💡 Note Imagine a relay team where each runner starts from wherever the last one happened to stop — chaos. Batch norm is the rule that every handoff happens at the same marked line. Runners can still choose their pace (that's γ and β), but nobody has to sprint across the field just to find the baton.
Related Questions
“Why do CNNs work so well for images?”
Answer
A fully connected network treats an image as a flat bag of pixels. Flatten a modest 224×224 color image and that's ~150,000 inputs; a single 1,000-unit hidden layer would need 150 million weights — for one layer. It's wasteful, it overfits, and it throws away spatial structure: an object learned in the top-left has to be re-learned in the bottom-right.
CNNs build the structure of images into the architecture with three ideas:
- Local connectivity. Each neuron connects only to a small patch (a 3×3 or 5×5 window), matching the fact that image features like edges and textures are local.
- Parameter sharing. The same small filter slides across every position, so a feature detector (say a vertical-edge detector) is learned once and reused everywhere. That's the big parameter cut — a filter is a few dozen weights, not millions.
- Translation invariance. Because the filter is shared and pooling summarizes local regions, the network recognizes a feature wherever it appears.
The real power comes from stacking these layers: early layers learn edges, middle layers combine them into textures and shapes, and deep layers assemble object parts and whole objects. Each layer's receptive field grows with depth, so the network builds global understanding out of local operations. That said, this locality bias is also a limitation — which is why Vision Transformers, using attention to relate distant patches, are competitive when you have enough data.
💡 Note A CNN filter is a rubber stamp. You carve the stamp once — say, an "edge" stamp — then press it all over the page to find every edge, instead of drawing a fresh detector by hand at each spot. Parameter sharing is reusing one stamp everywhere; local connectivity is the fact that the stamp is small.