Supervised Learning
Supervised Ensembles: Random Forests & Boosting
How multiple simple decision trees can collaborate to model complex boundaries without overfitting, and how boosting forces focus on difficult examples.
Stage 1 of 5: Single Weak Tree
A single depth-2 tree over 60 points in four alternating quadrants. It labels 70.0% of the training points correctly and 61.4% of 800 unseen points.
- Class A
- Class B
- Decision boundary
One shallow tree gets three splits, and the classes sit in four alternating quadrants. It fits 70% of the training points but reaches only 61% on data it has never seen — the honest baseline everything below has to beat.
Check your understanding
Combine multiple weak models to build a robust predictor, correcting errors iteratively. Observe the difference between Bagging (Random Forests) and Boosting.
What you are looking at
A dataset forming an XOR pattern—two classes arranged diagonally across four clusters. A single, simple decision tree (a "weak learner") is completely incapable of drawing a boundary that separates this data accurately. It can only make one horizontal or vertical split, leaving half the points misclassified.
Ensemble methods combine multiple weak learners into a single, highly accurate predictor. The two most common strategies are Bagging (Random Forests) and Boosting (AdaBoost).
Random Forests (Bagging)
Random Forests train dozens of independent trees in parallel. To ensure these trees are diverse, each one is trained on a random sample of the training data (a bootstrap sample). When making a prediction, the ensemble simply takes a majority vote from all its constituent trees.
- Diversity is key: If all trees were the same, their vote would not improve anything. The random sampling ensures that each tree focuses on slightly different aspects of the data.
- Robustness: Because individual trees overfit to different noise in their respective samples, averaging their predictions cancels out the noise, resulting in a smooth, robust decision boundary.
Boosting
While Random Forests train trees independently, Boosting trains them sequentially. Each new tree in the ensemble focuses specifically on correcting the mistakes made by the previous trees.
- Start with a weak model (like a decision stump).
- Evaluate its performance. Points that were misclassified are assigned a higher weight (represented visually by larger point sizes).
- Train the next model, forcing it to pay more attention to the highly weighted points.
- Repeat this process, continually shifting focus to the hardest examples.
The final prediction is a weighted sum of all the trees, where trees with lower error are given more influence.
Break it on purpose
Ensembles are powerful, but they handle bad data differently. Random Forests are highly robust against outliers because bagging averages out the noise. Boosting, on the other hand, is notoriously sensitive to outliers.
The vulnerability of Boosting
Because Boosting aggressively increases the weight of misclassified points round after round, a single extreme outlier can hijack the entire training process. The ensemble will warp its boundary and sacrifice general accuracy just to classify that one impossible point correctly.
Try it in the simulation: hover over a point to see how the trees in the Random Forest vote. Notice how the Random Forest boundary remains relatively stable. Then, switch to Boosting and watch the sample weights grow exponentially for the points the stumps struggle to classify. If one point was completely out of bounds, Boosting would dedicate all its trees to capturing it.
Reference
- Bootstrap resample
- Draw n rows with replacement from the n training rows; about 37% are left out of each tree
- Random forest
- Bagging plus a random subset of features considered at every split
- Majority vote
- ŷ(x) = mode{ h_t(x) : t = 1..T }
- AdaBoost vote weight
- α_t = ½·ln((1 − ε_t) / ε_t), where ε_t = Σ w_i · 1[h_t(x_i) ≠ y_i]
- AdaBoost reweighting
- w_i ← w_i · exp(−α_t) if correct, · exp(+α_t) if wrong, then renormalise so Σ w_i = 1
- Boosted prediction
- ŷ(x) = sign( Σ α_t · h_t(x) ), with h_t mapped to ±1
- Additive limit
- A sum of depth-1 stumps is f(x) + g(y) — no x·y interaction term, so it cannot express a checkerboard at any T
Break it on purpose
Drop the boosting base learner to depth 1 and watch training accuracy climb to 83% while held-out accuracy sits at 51% — a coin flip. A sum of depth-1 stumps has no term that depends on x and y together, so it can never express this checkerboard no matter how many rounds you add. It is memorising, not learning.