Ensembles: Voting, Stacking, and Blending
Voting combines predictions by a fixed rule; stacking learns a rule from prediction data. Either can combine models from the same family or from different families. We will start with an ensemble that loses to its strongest member, then examine how a learned combination is trained and evaluated. The central design question is which rows are used to train each stage.
In binary classification, hard voting counts predicted labels, while soft voting averages probabilities for the same class and then applies a decision threshold. Suppose three models give positive-class probabilities 0.9, 0.4, and 0.4. At threshold 0.5, hard voting chooses class 0 by two votes to one; soft voting chooses class 1 because the average is about 0.567. These rules use different information. Run the Python blocks in order; the later body examples reuse earlier imports and definitions. Printed values labelled cv are mean five-fold classification accuracies.
Equal-weight voting can make things worse
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier, VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import cross_val_score
X, y = make_classification(n_samples=4000, n_features=25, n_informative=12,
random_state=0)
members = lambda: [("rf", RandomForestClassifier(n_estimators=200, random_state=0)),
("lr", LogisticRegression(max_iter=2000)),
("nb", GaussianNB())]
for name, est in (("random forest", RandomForestClassifier(n_estimators=200,
random_state=0)),
("logistic", LogisticRegression(max_iter=2000)),
("naive bayes", GaussianNB())):
print(f"{name:24s} cv {cross_val_score(est, X, y, cv=5).mean():.4f}")
print(f"{'soft voting, all three':24s} "
f"cv {cross_val_score(VotingClassifier(members(), voting='soft'), X, y, cv=5).mean():.4f}")
print(f"{'hard voting, all three':24s} "
f"cv {cross_val_score(VotingClassifier(members(), voting='hard'), X, y, cv=5).mean():.4f}")
# random forest cv 0.9195
# logistic cv 0.7448
# naive bayes cv 0.7642
# soft voting, all three cv 0.8290
# hard voting, all three cv 0.8230The forest alone scores 0.9195. Combining it with two weaker models drops the ensemble to 0.8290 — nine points worse than the best member, and closer to the average of the three than to the maximum.
Equal weights give each probability estimate the same numerical influence. They do not adapt to the measured quality of the members. In this example the combination performs worse than the forest; accuracy alone does not tell us which individual disagreements or probability estimates caused the loss.
Voting can outperform the best member even when their accuracies differ. It can also underperform every member: if three classifiers each err on a different pair of rows among three rows, each is right once but the majority is wrong on all three. The joint pattern of predictions determines the result. Similar accuracy and complementary errors are useful things to investigate, not necessary-and-sufficient conditions or bounds on ensemble accuracy.
Learn the weights instead
A stacking meta-model takes base predictions as its input features. Here its input row has three positive-class probabilities, one each from the forest, logistic regression, and Naive Bayes. A logistic meta-model computes \(q=1/(1+e^{-s})\), where \(s=b+w_{\rm rf}p_{\rm rf}+w_{\rm lr}p_{\rm lr}+w_{\rm nb}p_{\rm nb}\). The intercept \(b\) and coefficients \(w\) are fitted. This is a logistic transformation of a learned score, so the coefficients are not normalized averaging weights.
import numpy as np
from sklearn.ensemble import StackingClassifier
weighted = VotingClassifier(members(), voting="soft", weights=[8, 1, 1])
stack = StackingClassifier(members(), final_estimator=LogisticRegression(), cv=5)
print(f"soft voting (equal) cv {cross_val_score(VotingClassifier(members(), voting='soft'), X, y, cv=5).mean():.4f}")
print(f"soft voting (weighted) cv {cross_val_score(weighted, X, y, cv=5).mean():.4f}")
print(f"stacking (learned) cv {cross_val_score(stack, X, y, cv=5).mean():.4f}")
stack.fit(X, y)
print(f"learned meta coefficients {np.round(stack.final_estimator_.coef_[0], 3)}"
f" (rf, lr, nb)")
print("meta intercept", round(stack.final_estimator_.intercept_[0], 3))
# soft voting (equal) cv 0.8290
# soft voting (weighted) cv 0.9025
# stacking (learned) cv 0.9293
# learned meta coefficients [12.832 -0.907 -1.42 ] (rf, lr, nb)
# meta intercept -5.184Stacking scores 0.9293 against the forest’s 0.9195 on the same outer folds, an observed gain of about 0.98 percentage points. This comparison describes the fixed candidates and split; selecting the best of many such attempts needs an additional evaluation. The printed coefficients, however, come from the separate final fit on all 4,000 rows; they are not the five outer-fold models’ coefficients. For this final fit, the coefficients are approximately 12.832, −0.907, and −1.420, with intercept −5.184. These numbers specify its score function; they do not isolate the cause of the cross-validation gain.
A negative coefficient means that increasing that member’s probability, while holding the other input probabilities fixed, lowers the meta-model’s positive-class score. Correlated inputs can receive such conditional corrections. The sign does not establish that the member is generally wrong, and negative coefficients are not required for an ensemble to beat its best member.
The weights \([8,1,1]\) make the forest contribute eight tenths of a soft-voting average. These are illustrative fixed weights. Selecting weights, base models, or meta-model settings using performance data is another search and needs evaluation outside that search. Stacking fits a combination by an optimization objective; it does not eliminate model selection.
How to construct the meta-model’s training inputs
Training the meta-model on base predictions from the same rows used to fit those bases can make it rely on training behavior that will not transfer. Out-of-fold (OOF) prediction avoids this direct reuse: divide the training rows into five folds, fit each base on four folds, and predict the remaining fold. Fill each row’s three prediction columns only when that row is held out. With 100 training rows, this produces a \(100\times3\) meta-input matrix from base fits on 80 rows each. The meta-model uses all 100 labels to learn the combination; deployment bases are then refitted on all 100 rows.
import numpy as np
from sklearn.model_selection import train_test_split, cross_val_predict
from sklearn.svm import SVC
from sklearn.calibration import CalibratedClassifierCV
rng = np.random.default_rng(0)
Xn = rng.normal(size=(1200, 40))
yn = rng.integers(0, 2, 1200) # no signal at all
A_tr, A_te, b_tr, b_te = train_test_split(Xn, yn, test_size=0.4, random_state=0)
bases = [RandomForestClassifier(n_estimators=100, random_state=0),
CalibratedClassifierCV(SVC(random_state=0), cv=3),
GaussianNB()]
Z_in = np.column_stack([m.fit(A_tr, b_tr).predict_proba(A_tr)[:, 1] for m in bases])
Z_te = np.column_stack([m.predict_proba(A_te)[:, 1] for m in bases])
in_sample = LogisticRegression().fit(Z_in, b_tr)
print(f"in-sample inputs: meta train {in_sample.score(Z_in, b_tr):.4f}"
f" test {in_sample.score(Z_te, b_te):.4f}")
Z_oof = np.column_stack([cross_val_predict(m, A_tr, b_tr, cv=5,
method="predict_proba")[:, 1]
for m in bases])
oof_meta = LogisticRegression().fit(Z_oof, b_tr)
print(f"OOF inputs: meta train {oof_meta.score(Z_oof, b_tr):.4f}"
f" test {oof_meta.score(Z_te, b_te):.4f}")
print("expected new-label accuracy: 0.5")
# in-sample inputs: meta train 1.0000 test 0.4979
# OOF inputs: meta train 0.5389 test 0.4667
# expected new-label accuracy: 0.5The labels were drawn independently of the features with probability 0.5. Conditional on any fitted classifier and new inputs, each independent new label is still a fair coin, so expected test accuracy is 0.5. A finite test score will fluctuate around it. The in-sample-input version scores 1.0000 on its meta-training rows and 0.4979 on the test set. The OOF-input version scores 0.5389 and 0.4667. The lower training score makes the lack of signal less easy to miss, but neither training score measures performance on new labels.
The score on Z_oof is also a meta-model training score: the logistic regression was fitted using those same rows and labels. OOF construction removes each row from its own base fit; it does not turn the fitted meta-model’s score into an independent evaluation. The untouched test rows evaluate the whole fitted procedure. On data with signal, in-sample and OOF training can also produce different generalization performance, so the distinction is more than a change in the training score.
The StackingClassifier documentation distinguishes inner prediction construction from evaluation. In the earlier example, StackingClassifier(cv=5) builds OOF inputs inside each outer training fold, while cross_val_score(..., cv=5) holds out rows from the entire stack. The binary probability inputs contribute one column per base. The cv="prefit" option skips OOF construction and needs separate attention to which rows trained the supplied estimators.
Keep learned preprocessing inside each base-model pipeline so it is refitted within the inner folds. Grouped data needs group-separated inner and outer splits. Time-ordered data needs past-to-future predictions; an ordinary expanding-window splitter leaves the earliest rows without held-out predictions and cannot simply be substituted into the partition-based cross_val_predict path. Build time-valid meta-inputs manually for rows with available historical training data, or use a suitable later holdout for blending.
Choosing between them
| Method | Combines by | What to assess |
|---|---|---|
| Hard voting | Votes over class labels | Error overlap and the decision rule, including ties |
| Soft voting | Average class probabilities | Probability quality and held-out performance of the average |
| Weighted voting | Prespecified or validation-selected weights | Whether weight selection improves performance outside its selection data |
| Stacking | A model fitted on OOF base predictions | Valid inner splits, meta-model complexity, and total fit cost |
| Blending | A model fitted on a separate holdout’s base predictions | Enough data for both stages and an independent final evaluation |
Soft voting can be useful with imperfectly calibrated members, but a confidently wrong member can pull the average across a decision threshold. Calibration is worth assessing when combining probability estimates, especially for models such as Naive Bayes that can be overconfident on dependent features. Fit any calibrator using data or internal folds separate from its underlying model’s fit, within the ensemble’s training procedure. Calibration may improve probability quality without improving classification accuracy, and the average itself is not guaranteed to be calibrated.
A stack requires several models at prediction time as well as during training. Compare its measured improvement with latency, memory, and maintenance costs. Models using different modalities or time windows may supply complementary information, and different fits to the same features can do so too. Evaluate the actual combination against a strong single-model baseline; this article’s examples do not establish a typical gain for production problems.
Exercises
1. When does voting help? Hold each member’s marginal accuracy fixed while changing how often their predictions use a shared random draw. Compare majority accuracy with the best member and the analytic expectation.
Read the measured individual accuracies before interpreting the correlation and voting columns. Which patterns improve on the strongest member?
Solution
import numpy as np
rng = np.random.default_rng(0)
n = 200_000
def make_members(accs, shared):
"""shared is the probability of using one common uniform draw."""
accs = np.asarray(accs)
coupled = rng.random((n, 1)) < shared
common = rng.random((n, 1))
independent = rng.random((n, len(accs)))
return np.where(coupled, common, independent) < accs
for shared in (0.0, 0.8, 1.0):
for accs in ([0.75, 0.75, 0.75], [0.85, 0.80, 0.80],
[0.90, 0.75, 0.75], [0.90, 0.60, 0.60]):
correct = make_members(accs, shared)
measured = correct.mean(axis=0)
vote = (correct.sum(axis=1) >= 2).mean()
corr = np.corrcoef((~correct).T)[np.triu_indices(3, 1)].mean()
a, b, c = accs
independent_vote = a*b + a*c + b*c - 2*a*b*c
expected = (1-shared)*independent_vote + shared*np.median(accs)
print(f"shared {shared:.1f} targets {accs} measured {np.round(measured, 3)}"
f" mean error corr {corr:.3f} vote {vote:.4f} expected {expected:.4f}")
# shared 0.0 targets [0.75, 0.75, 0.75] measured [0.749 0.751 0.749] mean error corr -0.000 vote 0.8436 expected 0.8438
# shared 0.0 targets [0.85, 0.8, 0.8] measured [0.851 0.799 0.799] mean error corr -0.000 vote 0.9118 expected 0.9120
# shared 0.0 targets [0.9, 0.75, 0.75] measured [0.9 0.752 0.75 ] mean error corr 0.001 vote 0.9002 expected 0.9000
# shared 0.0 targets [0.9, 0.6, 0.6] measured [0.899 0.6 0.601] mean error corr 0.001 vote 0.7920 expected 0.7920
# shared 0.8 targets [0.75, 0.75, 0.75] measured [0.75 0.75 0.75] mean error corr 0.803 vote 0.7685 expected 0.7688
# shared 0.8 targets [0.85, 0.8, 0.8] measured [0.851 0.801 0.801] mean error corr 0.719 vote 0.8234 expected 0.8224
# shared 0.8 targets [0.9, 0.75, 0.75] measured [0.9 0.75 0.75] mean error corr 0.572 vote 0.7802 expected 0.7800
# shared 0.8 targets [0.9, 0.6, 0.6] measured [0.9 0.6 0.6] mean error corr 0.483 vote 0.6381 expected 0.6384
# shared 1.0 targets [0.75, 0.75, 0.75] measured [0.753 0.753 0.753] mean error corr 1.000 vote 0.7528 expected 0.7500
# shared 1.0 targets [0.85, 0.8, 0.8] measured [0.851 0.801 0.801] mean error corr 0.893 vote 0.8010 expected 0.8000
# shared 1.0 targets [0.9, 0.75, 0.75] measured [0.9 0.751 0.751] mean error corr 0.719 vote 0.7511 expected 0.7500
# shared 1.0 targets [0.9, 0.6, 0.6] measured [0.9 0.599 0.599] mean error corr 0.606 vote 0.5993 expected 0.6000Each Boolean entry records whether a member is correct on a binary classification case. With probability shared, all members compare the same uniform draw with their target accuracies; otherwise each uses its own draw. Either draw is uniform, so a member with target accuracy \(a\) remains correct with probability \(a\). The shared parameter is a mixture probability, not an error correlation or the fraction of all errors shared.
For three independent correctness indicators with probabilities \(a,b,c\), majority accuracy is \(ab+ac+bc-2abc\). For equal accuracy 0.75, this is \(3(0.75)^2-2(0.75)^3=0.84375\). In the fully coupled case, the middle of the three target accuracies determines whether at least two members are correct. Mixing the two mechanisms gives the expected column. For odd ensembles of independent voters with the same accuracy above 0.5, the classical majority-vote result predicts improvement as the number of voters grows.
The unequal accuracies 0.85, 0.80, and 0.80 give independent majority accuracy 0.912, above the best member’s 0.85. The combination 0.90, 0.60, and 0.60 instead gives 0.792, below 0.90. Unequal strength therefore changes the calculation without deciding its outcome by itself. Shared errors matter through their joint distribution; the variance floor for an average of real-valued predictions is not a formula for majority-vote classification error.
High pairwise error correlation also supplies no universal cutoff: with equal 0.75 accuracies and shared=0.8, error correlation is 0.8 in the population, yet expected majority accuracy is \(0.8(0.75)+0.2(0.84375)=0.76875\), still above 0.75. Marginal accuracies and pairwise correlations alone need not determine the joint behavior of a larger ensemble. Compare candidate combinations on validation data instead of applying a fixed correlation threshold.
2. How much data do the two stages use? Compare base models, stacking, and blending on increasing prefixes of one training pool, using a fixed held-out set for this learning-curve experiment.
Track the number of rows used by each stage and compare the scores without assuming that stacking must win above a particular sample size.
Solution
import numpy as np
from sklearn.base import clone
from sklearn.datasets import make_classification
from sklearn.ensemble import (RandomForestClassifier, StackingClassifier,
HistGradientBoostingClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = make_classification(n_samples=10_000, n_features=25, n_informative=12,
random_state=0)
X_pool, X_test, y_pool, y_test = train_test_split(
X, y, test_size=2000, random_state=0, stratify=y)
members = [("rf", RandomForestClassifier(n_estimators=100, random_state=0)),
("gb", HistGradientBoostingClassifier(random_state=0)),
("lr", make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)))]
for n in (200, 600, 2000, 8000):
X_train, y_train = X_pool[:n], y_pool[:n]
base_scores = [clone(m).fit(X_train, y_train).score(X_test, y_test)
for _, m in members]
stack = StackingClassifier(members, final_estimator=LogisticRegression(), cv=5)
stack.fit(X_train, y_train)
X_base, X_meta, y_base, y_meta = train_test_split(
X_train, y_train, test_size=0.25, random_state=0, stratify=y_train)
fitted = [clone(m).fit(X_base, y_base) for _, m in members]
Z_meta = np.column_stack([m.predict_proba(X_meta)[:, 1] for m in fitted])
Z_test = np.column_stack([m.predict_proba(X_test)[:, 1] for m in fitted])
blender = LogisticRegression().fit(Z_meta, y_meta)
print(f"n={n:5d} best observed base {max(base_scores):.4f}"
f" stack {stack.score(X_test, y_test):.4f}"
f" blend {blender.score(Z_test, y_test):.4f}")
# n= 200 best observed base 0.7405 stack 0.7420 blend 0.7275
# n= 600 best observed base 0.8690 stack 0.8735 blend 0.8530
# n= 2000 best observed base 0.9290 stack 0.9320 blend 0.9195
# n= 8000 best observed base 0.9440 stack 0.9515 blend 0.9440All training sizes come from one generated dataset, so the feature-to-label relation and held-out rows stay fixed. The stack’s meta-input matrix has \(n\) rows and three columns; its logistic meta-model fits three coefficients and an intercept, with regularization. Each five-fold base fit uses about \(0.8n\) rows. Here stacking already slightly exceeds the best observed base at 200 rows (0.7420 versus 0.7405, three extra correct predictions on the 2,000-row test set). At 8,000 rows the scores are 0.9515 and 0.9440. More rows can stabilize both stages, but these single-run differences do not establish a minimum sample size or a reliable gain.
The OOF construction fits each base five times, and the deployed version is fitted once on the full training subset: six fits per base, or 18 base fits for this three-member stack, plus the meta-model. The smaller OOF fits can produce predictions distributed differently from those of the final refits; whether that mismatch harms performance depends on the learners and the sample size.
The blender fits bases on 75% of the current training subset and its meta-model on predictions for the remaining 25%. At \(n=200\), those stages use 150 and 50 rows; at \(n=8000\), they use 6,000 and 2,000. It keeps these fitted bases for test prediction, making three base fits plus one meta fit. Refitting the bases on additional rows afterward changes the inputs the blender will receive and should be evaluated as a separate procedure.
The “best observed base” is the maximum held-out score among three fitted candidates, a descriptive comparator chosen after seeing these results. It is not an independently evaluated selection procedure. If the learning curves guide a choice, use another test set for the final assessment. A regularized linear meta-model is a useful starting point; more flexible alternatives can be evaluated with an outer split rather than ruled out by a universal row-count threshold.
3. What disagreement tells you. Measure disagreement and shared errors for each pair, then evaluate that same pair’s soft-voting combination.
Compare the pair’s observed score with its stronger member. Can the two label-based diagnostics alone account for all the changes?
Solution
import numpy as np
from itertools import combinations
from sklearn.datasets import make_classification
from sklearn.ensemble import (RandomForestClassifier, ExtraTreesClassifier,
HistGradientBoostingClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=6000, n_features=25, n_informative=12,
random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.4, random_state=0)
cands = {"rf": RandomForestClassifier(n_estimators=200, random_state=0),
"et": ExtraTreesClassifier(n_estimators=200, random_state=0),
"gb": HistGradientBoostingClassifier(random_state=0),
"lr": LogisticRegression(max_iter=2000),
"kn": KNeighborsClassifier(15)}
proba = {k: m.fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
for k, m in cands.items()}
pred = {k: p > 0.5 for k, p in proba.items()}
acc = {k: (p == y_te).mean() for k, p in pred.items()}
print("individual:", {k: float(f"{v:.4f}") for k, v in acc.items()})
for a, b in combinations(cands, 2):
disagree = (pred[a] != pred[b]).mean()
both_wrong = ((pred[a] != y_te) & (pred[b] != y_te)).mean()
vote = (((proba[a] + proba[b]) / 2 > 0.5) == y_te).mean()
print(f"{a}+{b} disagree {disagree:.4f} both wrong {both_wrong:.4f}"
f" best {max(acc[a], acc[b]):.4f} pair soft vote {vote:.4f}")
# individual: {'rf': 0.9329, 'et': 0.9396, 'gb': 0.9463, 'lr': 0.715, 'kn': 0.9517}
# rf+et disagree 0.0283 both wrong 0.0496 best 0.9396 pair soft vote 0.9392
# rf+gb disagree 0.0333 both wrong 0.0437 best 0.9463 pair soft vote 0.9437
# rf+lr disagree 0.2571 both wrong 0.0475 best 0.9329 pair soft vote 0.8642
# rf+kn disagree 0.0504 both wrong 0.0325 best 0.9517 pair soft vote 0.9471
# et+gb disagree 0.0375 both wrong 0.0383 best 0.9463 pair soft vote 0.9483
# et+lr disagree 0.2621 both wrong 0.0417 best 0.9396 pair soft vote 0.8600
# et+kn disagree 0.0379 both wrong 0.0354 best 0.9517 pair soft vote 0.9479
# gb+lr disagree 0.2646 both wrong 0.0371 best 0.9463 pair soft vote 0.9300
# gb+kn disagree 0.0488 both wrong 0.0267 best 0.9517 pair soft vote 0.9525
# lr+kn disagree 0.2750 both wrong 0.0292 best 0.9517 pair soft vote 0.9313Disagreement counts rows where the two thresholded predictions differ. Both-wrong counts their shared mistakes. In binary classification with a common 0.5 threshold, if both members predict the same wrong label, their equally weighted probability average stays on that wrong side too. Thus \(1-\text{both-wrong}\) is an upper bound on this pair’s accuracy, not a score the average necessarily achieves.
On a disagreement, the average’s decision depends on the probability magnitudes. If class 1 is correct, probabilities 0.9 and 0.4 average to 0.65 and succeed; probabilities 0.6 and 0.1 average to 0.35 and fail. Both cases have the same hard-label disagreement. This is why label-based diversity summaries do not determine a soft-voting gain. If disagreement is exactly zero, the pair average cannot change any decision under the common threshold rule used here.
The code fits each candidate once and reuses its probability column for every pair, so each reported ensemble contains exactly the two named members. A model-family label is a reason to test a combination, not a measurement of complementary errors. For example, rf+lr disagrees on 25.71% of rows but its soft vote scores 0.8642, below the forest’s 0.9329. The less-disagreeing et+gb pair scores 0.9483 against its best member’s 0.9463, five additional correct decisions on 2,400 rows. Larger disagreement alone did not predict the better combination.
These diagnostics help explain a measured combination and identify candidates for further validation. If this held-out split is used to select the pair, it becomes selection data; report the chosen pair’s final performance on new held-out data. Prediction columns and class labels must also be aligned before any averaging.
References
- Wolpert, D. H. (1992). Stacked Generalization. Neural Networks.
- Breiman, L. (1996). Stacked Regressions. Machine Learning.
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.