Voting, Stacking, and Blending Ensembles

Voting combines predictions by a fixed rule; stacking learns a rule from prediction data. Either can combine models from the same family or from different families. We will start with an ensemble that loses to its strongest member, then examine how a learned combination is trained and evaluated. The central design question is which rows are used to train each stage.

In binary classification, hard voting counts predicted labels, while soft voting averages probabilities for the same class and then applies a decision threshold. Suppose three models give positive-class probabilities 0.9, 0.4, and 0.4. At threshold 0.5, hard voting chooses class 0 by two votes to one; soft voting chooses class 1 because the average is about 0.567. These rules use different information. Run the Python blocks in order; the later body examples reuse earlier imports and definitions. Printed values labeled cv are mean five-fold classification accuracies.

Equal-weight voting can make things worse

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier, VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import cross_val_score

X, y = make_classification(n_samples=4000, n_features=25, n_informative=12,
                           random_state=0)
members = lambda: [("rf", RandomForestClassifier(n_estimators=200, random_state=0)),
                   ("lr", LogisticRegression(max_iter=2000)),
                   ("nb", GaussianNB())]

for name, est in (("random forest", RandomForestClassifier(n_estimators=200,
                                                           random_state=0)),
                  ("logistic", LogisticRegression(max_iter=2000)),
                  ("naive bayes", GaussianNB())):
    print(f"{name:24s} cv {cross_val_score(est, X, y, cv=5).mean():.4f}")

print(f"{'soft voting, all three':24s} "
      f"cv {cross_val_score(VotingClassifier(members(), voting='soft'), X, y, cv=5).mean():.4f}")
print(f"{'hard voting, all three':24s} "
      f"cv {cross_val_score(VotingClassifier(members(), voting='hard'), X, y, cv=5).mean():.4f}")
# random forest            cv 0.9195
# logistic                 cv 0.7448
# naive bayes              cv 0.7642
# soft voting, all three   cv 0.8290
# hard voting, all three   cv 0.8230

The forest alone scores 0.9195. Combining it with two weaker models drops the ensemble to 0.8290 — nine points worse than the best member, and closer to the average of the three than to the maximum.

Equal weights give each probability estimate the same numerical influence. They do not adapt to the measured quality of the members. In this example the combination performs worse than the forest; accuracy alone does not tell us which individual disagreements or probability estimates caused the loss.

Voting can outperform the best member even when their accuracies differ. It can also underperform every member: if three classifiers each err on a different pair of rows among three rows, each is right once but the majority is wrong on all three. The joint pattern of predictions determines the result. Similar accuracy and complementary errors are useful things to investigate, not necessary-and-sufficient conditions or bounds on ensemble accuracy.

Learn the weights instead

A stacking meta-model takes base predictions as its input features. Here its input row has three positive-class probabilities, one each from the forest, logistic regression, and Naive Bayes. A logistic meta-model computes \(q=1/(1+e^{-s})\), where \(s=b+w_{\rm rf}p_{\rm rf}+w_{\rm lr}p_{\rm lr}+w_{\rm nb}p_{\rm nb}\). The intercept \(b\) and coefficients \(w\) are fitted. This is a logistic transformation of a learned score, so the coefficients are not normalized averaging weights.

import numpy as np
from sklearn.ensemble import StackingClassifier

weighted = VotingClassifier(members(), voting="soft", weights=[8, 1, 1])
stack = StackingClassifier(members(), final_estimator=LogisticRegression(), cv=5)

print(f"soft voting (equal)     cv {cross_val_score(VotingClassifier(members(), voting='soft'), X, y, cv=5).mean():.4f}")
print(f"soft voting (weighted)  cv {cross_val_score(weighted, X, y, cv=5).mean():.4f}")
print(f"stacking (learned)      cv {cross_val_score(stack, X, y, cv=5).mean():.4f}")
stack.fit(X, y)
print(f"learned meta coefficients {np.round(stack.final_estimator_.coef_[0], 3)}"
      f"  (rf, lr, nb)")
print("meta intercept", round(stack.final_estimator_.intercept_[0], 3))
# soft voting (equal)     cv 0.8290
# soft voting (weighted)  cv 0.9025
# stacking (learned)      cv 0.9293
# learned meta coefficients [12.832 -0.907 -1.42 ]  (rf, lr, nb)
# meta intercept -5.184

Stacking scores 0.9293 against the forest’s 0.9195 on the same outer folds, an observed gain of about 0.98 percentage points. This comparison describes the fixed candidates and split; selecting the best of many such attempts needs an additional evaluation. The printed coefficients, however, come from the separate final fit on all 4,000 rows; they are not the five outer-fold models’ coefficients. For this final fit, the coefficients are approximately 12.832, −0.907, and −1.420, with intercept −5.184. These numbers specify its score function; they do not isolate the cause of the cross-validation gain.

A negative coefficient means that increasing that member’s probability, while holding the other input probabilities fixed, lowers the meta-model’s positive-class score. Correlated inputs can receive such conditional corrections. The sign does not establish that the member is generally wrong, and negative coefficients are not required for an ensemble to beat its best member.

The weights \([8,1,1]\) make the forest contribute eight tenths of a soft-voting average. These are illustrative fixed weights. Selecting weights, base models, or meta-model settings using performance data is another search and needs evaluation outside that search. Stacking fits a combination by an optimization objective; it does not eliminate model selection.

How to construct the meta-model’s training inputs

Training the meta-model on base predictions from the same rows used to fit those bases can make it rely on training behavior that will not transfer. Out-of-fold (OOF) prediction avoids this direct reuse: divide the training rows into five folds, fit each base on four folds, and predict the remaining fold. Fill each row’s three prediction columns only when that row is held out. With 100 training rows, this produces a \(100\times3\) meta-input matrix from base fits on 80 rows each. The meta-model uses all 100 labels to learn the combination; deployment bases are then refitted on all 100 rows.

import numpy as np
from sklearn.model_selection import train_test_split, cross_val_predict
from sklearn.svm import SVC
from sklearn.calibration import CalibratedClassifierCV

rng = np.random.default_rng(0)
Xn = rng.normal(size=(1200, 40))
yn = rng.integers(0, 2, 1200)                  # no signal at all
A_tr, A_te, b_tr, b_te = train_test_split(Xn, yn, test_size=0.4, random_state=0)
bases = [RandomForestClassifier(n_estimators=100, random_state=0),
         CalibratedClassifierCV(SVC(random_state=0), cv=3),
         GaussianNB()]

Z_in = np.column_stack([m.fit(A_tr, b_tr).predict_proba(A_tr)[:, 1] for m in bases])
Z_te = np.column_stack([m.predict_proba(A_te)[:, 1] for m in bases])
in_sample = LogisticRegression().fit(Z_in, b_tr)
print(f"in-sample inputs:   meta train {in_sample.score(Z_in, b_tr):.4f}"
      f"   test {in_sample.score(Z_te, b_te):.4f}")

Z_oof = np.column_stack([cross_val_predict(m, A_tr, b_tr, cv=5,
                                           method="predict_proba")[:, 1]
                         for m in bases])
oof_meta = LogisticRegression().fit(Z_oof, b_tr)
print(f"OOF inputs:         meta train {oof_meta.score(Z_oof, b_tr):.4f}"
      f"   test {oof_meta.score(Z_te, b_te):.4f}")
print("expected new-label accuracy: 0.5")
# in-sample inputs:   meta train 1.0000   test 0.4979
# OOF inputs:         meta train 0.5389   test 0.4667
# expected new-label accuracy: 0.5

The labels were drawn independently of the features with probability 0.5. Conditional on any fitted classifier and new inputs, each independent new label is still a fair coin, so expected test accuracy is 0.5. A finite test score will fluctuate around it. The in-sample-input version scores 1.0000 on its meta-training rows and 0.4979 on the test set. The OOF-input version scores 0.5389 and 0.4667. The lower training score makes the lack of signal less easy to miss, but neither training score measures performance on new labels.

The score on Z_oof is also a meta-model training score: the logistic regression was fitted using those same rows and labels. OOF construction removes each row from its own base fit; it does not turn the fitted meta-model’s score into an independent evaluation. The untouched test rows evaluate the whole fitted procedure. On data with signal, in-sample and OOF training can also produce different generalization performance, so the distinction is more than a change in the training score.

The StackingClassifier documentation distinguishes inner prediction construction from evaluation. In the earlier example, StackingClassifier(cv=5) builds OOF inputs inside each outer training fold, while cross_val_score(..., cv=5) holds out rows from the entire stack. The binary probability inputs contribute one column per base. The cv="prefit" option skips OOF construction and needs separate attention to which rows trained the supplied estimators.

Keep learned preprocessing inside each base-model pipeline so it is refitted within the inner folds. Grouped data needs group-separated inner and outer splits. Time-ordered data needs past-to-future predictions; an ordinary expanding-window splitter leaves the earliest rows without held-out predictions and cannot simply be substituted into the partition-based cross_val_predict path. Build time-valid meta-inputs manually for rows with available historical training data, or use a suitable later holdout for blending.

Choosing between them

Method	Combines by	What to assess
Hard voting	Votes over class labels	Error overlap and the decision rule, including ties
Soft voting	Average class probabilities	Probability quality and held-out performance of the average
Weighted voting	Prespecified or validation-selected weights	Whether weight selection improves performance outside its selection data
Stacking	A model fitted on OOF base predictions	Valid inner splits, meta-model complexity, and total fit cost
Blending	A model fitted on a separate holdout’s base predictions	Enough data for both stages and an independent final evaluation

Soft voting can be useful with imperfectly calibrated members, but a confidently wrong member can pull the average across a decision threshold. Calibration is worth assessing when combining probability estimates, especially for models such as Naive Bayes that can be overconfident on dependent features. Fit any calibrator using data or internal folds separate from its underlying model’s fit, within the ensemble’s training procedure. Calibration may improve probability quality without improving classification accuracy, and the average itself is not guaranteed to be calibrated.

A stack requires several models at prediction time as well as during training. Compare its measured improvement with latency, memory, and maintenance costs. Models using different modalities or time windows may supply complementary information, and different fits to the same features can do so too. Evaluate the actual combination against a strong single-model baseline; this article’s examples do not establish a typical gain for production problems.

References

• Wolpert, D. H. (1992). Stacked Generalization. Neural Networks.
• Breiman, L. (1996). Stacked Regressions. Machine Learning.
• scikit-learn documentation. sklearn.ensemble.StackingClassifier