Bagging vs Boosting vs Stacking

The Short Answer

Ensemble learning relies on the "wisdom of the crowd." Instead of trusting one model, you build multiple models and combine their outputs.

• Bagging (Bootstrap Aggregating): Trains many identical but independent models on random subsets of the data in parallel, then averages their votes. Example: Random Forest.
• Boosting: Trains many identical models sequentially. Each new model specifically focuses on fixing the mistakes made by the previous model. Example: XGBoost.
• Stacking: Trains completely different types of models (e.g., a Tree, an SVM, and a Neural Net) and then uses a final "meta-model" to figure out which model to trust for which inputs.

Where They Differ

Feature	Bagging	Boosting	Stacking
Model Types	Homogeneous (same type, usually Decision Trees).	Homogeneous (usually shallow Decision Trees).	Heterogeneous (mix of completely different algorithms).
Training Execution	Parallel (fast).	Sequential (slower).	Parallel base models, then sequential meta-model.
Primary Goal	Reduce Variance (prevent overfitting).	Reduce Bias (prevent underfitting).	Maximize overall predictive power.
Handling Mistakes	Averages them out.	Learns from them aggressively.	Learns who makes what mistakes.

Choose Bagging When

• You have a high-variance model: You are using deep, unpruned decision trees that memorize the training data.
• You need it fast: Because the models are independent, you can train 1,000 trees simultaneously across 1,000 CPU cores.

Choose Boosting When

• You need state-of-the-art accuracy on tabular data: Boosting algorithms (like LightGBM, CatBoost, and XGBoost) dominate competitions for structured data.
• You have high bias: Your base model is too weak to capture the patterns. Boosting forces the ensemble to become highly complex.
• Warning: Because boosting constantly attacks errors, it is prone to overfitting if you let it run too long or if your data is very noisy (it will try to fit the noise).

Choose Stacking When

• You are trying to win a Kaggle competition: Stacking almost always squeezes out the last few drops of accuracy.
• Your models have different strengths: You have one model that is great at finding linear relationships, and another that is great at finding non-linear relationships. Stacking lets a meta-model learn exactly when to trust which base model.
• Warning: Stacking is notoriously difficult to maintain in production. You have to maintain, deploy, and monitor 4 different models instead of 1.

What People Get Wrong

Thinking Random Forest is just 'many trees'

A Random Forest isn't just an ensemble of trees; it's a specific implementation of Bagging. It relies on Bootstrap Aggregating (randomly sampling rows) AND feature randomness (randomly sampling columns at each split) to ensure the trees are mathematically de-correlated. If the trees all vote exactly the same way, the ensemble is useless.