Recent works demonstrated the existence of a double-descent phenomenon for the generalization error of neural networks, where highly overparameterized models escape overfitting and achieve good test performance, at odds with the standard bias-variance trade-off described by statistical learning theory. In the present work, we explore a link between this phenomenon and the increase of complexity and sensitivity of the function represented by neural networks. In particular, we study the Boolean mean dimension (BMD), a metric developed in the context of Boolean function analysis. Focusing on a simple teacher-student setting for the random feature model, we derive a theoretical analysis based on the replica method that yields an interpretable expression for the BMD, in the high dimensional regime where the number of data points, the number of features, and the input size grow to infinity. We find that, as the degree of overparameterization of the network is increased, the BMD reaches an evident peak at the interpolation threshold, in correspondence with the generalization error peak, and then slowly approaches a low asymptotic value. The same phenomenology is then traced in numerical experiments with different model classes and training setups. Moreover, we find empirically that adversarially initialized models tend to show higher BMD values, and that models that are more robust to adversarial attacks exhibit a lower BMD.

翻译：近期研究揭示了神经网络泛化误差存在“双下降”现象：高度过参数化模型能够规避过拟合，并获得优异的测试性能，这一现象与统计学习理论中标准的偏差-方差权衡相悖。本研究探索了该现象与神经网络所表征函数的复杂度及敏感性增长之间的关联。具体而言，我们研究了布尔均值维度（BMD）这一在布尔函数分析领域发展的度量指标。针对随机特征模型的简单师生学习框架，我们基于副本方法进行了理论分析，推导出高维极限（数据点数量、特征维度与输入规模同步趋于无穷大）下BMD的可解释表达式。研究发现：随着网络过参数化程度的提升，BMD在插值阈值处达到显著峰值（对应于泛化误差峰值），随后缓慢趋近于较低的渐近值。不同模型类别与训练配置下的数值实验均复现了相同现象。此外，实证结果表明：对抗性初始化的模型倾向于呈现更高BMD值，而对对抗攻击具有更强鲁棒性的模型则表现出更低的BMD。