Deep, overparameterized regression models are notorious for their tendency to overfit. This problem is exacerbated in heteroskedastic models, which predict both mean and residual noise for each data point. At one extreme, these models fit all training data perfectly, eliminating residual noise entirely; at the other, they overfit the residual noise while predicting a constant, uninformative mean. We observe a lack of middle ground, suggesting a phase transition dependent on model regularization strength. Empirical verification supports this conjecture by fitting numerous models with varying mean and variance regularization. To explain the transition, we develop a theoretical framework based on a statistical field theory, yielding qualitative agreement with experiments. As a practical consequence, our analysis simplifies hyperparameter tuning from a two-dimensional to a one-dimensional search, substantially reducing the computational burden. Experiments on diverse datasets, including UCI datasets and the large-scale ClimSim climate dataset, demonstrate significantly improved performance in various calibration tasks.

翻译：深度过参数化回归模型以其过度拟合的倾向而臭名昭著。这一问题在异方差模型中进一步加剧，该类模型同时预测每个数据点的均值和残差噪声。在极端情况下，这类模型要么完美拟合所有训练数据，完全消除残差噪声；要么过度拟合残差噪声，同时预测恒定且无信息的均值。我们观察到缺乏中间地带，这表明存在一个依赖于模型正则化强度的相变。通过拟合大量具有不同均值和方差正则化的模型，经验验证支持了这一猜想。为解释这一相变，我们基于统计场论发展了一个理论框架，与实验定性一致。作为实际结果，我们的分析将超参数调优从二维搜索简化为单维搜索，大大降低了计算负担。在包括UCI数据集和大规模ClimSim气候数据集在内的多样数据集上的实验表明，在各类校准任务中性能显著提升。