$\nu$-ensembles: Improving deep ensemble calibration in the small data regime

TL;DR — We use unlabeled data to improve deep ensemble diversity and calibration for small to medium-sized training sets.

Abstract

We present a method to improve the calibration of deep ensembles in the small data regime in the presence of unlabeled data. Our approach, which we name $\nu$-ensembles, is extremely easy to implement: given an unlabeled set, for each unlabeled data point, we simply fit a different randomly selected label with each ensemble member. We provide a theoretical analysis based on a PAC-Bayes bound which guarantees that for such a labeling we obtain low negative log-likelihood and high ensemble diversity on testing samples. Empirically, through detailed experiments, we find that for low to moderately-sized training sets, $\nu$-ensembles are more diverse and provide better calibration than standard ensembles, sometimes significantly.