Classical wisdom in machine learning holds that the generalization error can be decomposed into bias and variance, and these two terms exhibit a \emph{trade-off}. However, in this paper, we show that for an ensemble of deep learning based classification models, bias and variance are \emph{aligned} at a sample level, where squared bias is approximately \emph{equal} to variance for correctly classified sample points. We present empirical evidence confirming this phenomenon in a variety of deep learning models and datasets. Moreover, we study this phenomenon from two theoretical perspectives: calibration and neural collapse. We first show theoretically that under the assumption that the models are well calibrated, we can observe the bias-variance alignment. Second, starting from the picture provided by the neural collapse theory, we show an approximate correlation between bias and variance.

翻译：机器学习中的经典观点认为，泛化误差可分解为偏差与方差，且这两项之间存在权衡关系。然而，本文表明，对于基于深度学习的分类模型集成而言，偏差与方差在样本层面是"对齐"的——对于正确分类的样本点，平方偏差近似等于方差。我们通过多种深度学习模型与数据集提供了经验证据证实这一现象。此外，我们从两个理论视角研究该现象：校准与神经坍缩。首先在理论上证明，若假设模型是良好校准的，则可观察到偏差-方差对齐现象。其次，从神经坍缩理论提供的框架出发，我们展示了偏差与方差之间的近似相关性。