Bagging vs Boosting

Individual decision trees fail in one of two distinct ways. Fully grown, unconstrained trees exhibit high variance: their decision boundaries fluctuate wildly in response to small training perturbations, overfitting sample noise. In contrast, shallow decision stumps exhibit high bias: their rigid single-split axis cuts underfit the problem, persistently misclassifying non-linear class distributions. Ensemble algorithms resolve this trade-off by combining collections of learners.

Deal Overlapping Hands

Bootstrap Aggregation (Bagging) reduces variance through parallel randomization. Drawing $N$ instances with replacement from a dataset of size $N$ generates overlapping bootstrap folds, each containing approximately $63.2\%$ unique samples while leaving $36.8\%$ out-of-bag. Multiple deep decision trees are trained independently and in parallel across these distinct data views, with no communication between models.

Chase the Leftover

Boosting reduces bias through sequential error correction. Rather than training models independently, each base learner directly attacks the errors made by its predecessors. When the true label is $y = 5.0$ and the initial model predicts $\hat{y}_1 = 2.0$, the leftover residual is $r_1 = y - \hat{y}_1 = 3.0$. The subsequent tree is trained with this residual $3.0$ as its explicit optimization target, gradually refining the global prediction.

Combine the Votes

The two paradigms aggregate predictions through fundamentally different voting rules. Bagging treats all ensemble members as equal peers: individual classifications $(1, 0, 1)$ are averaged uniformly as $\frac{1 + 0 + 1}{3} = 0.67$. Boosting computes a linear combination of stage-weighted predictors $F(x) = \sum_{m=1}^M \alpha_m h_m(x)$, where highly accurate base models earn larger voting coefficients: $+0.42 - 0.42 + 0.65 = +0.65$.

Calmer and Sharper

Both strategies produce dramatic empirical improvements on identical datasets. For correlated trees with pairwise correlation $\rho = 0.4$, averaging 5 parallel trees cuts prediction variance from $1.00$ down to $0.40 \times 1.0 + \frac{0.60}{5} \times 1.0 = 0.52$. In parallel, chaining 20 boosted stumps drives cumulative residual bias down from $3.0$ toward $0.2$, turning blunt weak learners into sharp boundary sculptors.

Choose by the Data

Selecting between bagging and boosting depends on dataset noise characteristics. When datasets contain noisy labels or measurement contamination, choose Bagging: parallel averaging washes out random label errors. When labels are reliable and clean but model capacity lags behind a complex signal, choose Boosting: iterative residual optimization carves intricate non-linear frontiers.

Where It Breaks

Sequential residual fitting creates a critical vulnerability when confronted with mislabeled outliers. In Bagging, an erroneous point only appears in a subset of bootstrap samples and is easily outvoted by the honest majority. In Boosting, an uncorrectable mislabeled sample generates persistent, massive residuals; successive boosting rounds assign it escalating weights, ultimately warping the decision boundary into an unnatural loop around the corrupted dot.

The Quick Version

• Deep trees suffer high variance; shallow stumps suffer high bias.
• Bootstrap sampling deals overlapping subsets with replacement.
• Sequential boosting trains each base tree on prior residuals.
• Bagging averages equal votes; boosting sums earned stage weights.
• Averaging cuts variance to 0.52; chaining reduces bias to 0.2.
• Bagging cleans noisy labels; boosting sculpts intricate signals.
• Mislabeled outliers warp sequential boosting boundaries into loops.