A single model has blind spots. Ensemble methods train several models and merge their decisions, so the errors one learner makes are corrected by the others. The result is usually more accurate and more stable than any individual model.
Random forest — a grove of trees
- Grows many decision trees, each on a random subset of the data
- Reaches a final decision by majority vote
- Robust against overfitting and reveals which features matter most
Boosting — learning from mistakes
- AdaBoost re-weights misclassified cases each round so weak classifiers combine into a strong one
- Gradient boosting, as in XGBoost, corrects previous errors by optimising a loss function
- Powerful and handles missing values, but needs careful tuning
On breast-tumour data, random forests and boosted trees match or exceed carefully tuned single models — and produce fewer false positives for cancer.