Recent works demonstrated the existence of a double-descent phenomenon for the generalization error of neural networks, where highly overparameterized models escape overfitting and achieve good test performance, at odds with the standard bias-variance trade-off described by statistical learning theory. In the present work, we explore a link between this phenomenon and the increase of complexity and sensitivity of the function represented by neural networks. In particular, we study the Boolean mean dimension (BMD), a metric developed in the context of Boolean function analysis. Focusing on a simple teacher-student setting for the random feature model, we derive a theoretical analysis based on the replica method that yields an interpretable expression for the BMD, in the high dimensional regime where the number of data points, the number of features, and the input size grow to infinity. We find that, as the degree of overparameterization of the network is increased, the BMD reaches an evident peak at the interpolation threshold, in correspondence with the generalization error peak, and then slowly approaches a low asymptotic value. The same phenomenology is then traced in numerical experiments with different model classes and training setups. Moreover, we find empirically that adversarially initialized models tend to show higher BMD values, and that models that are more robust to adversarial attacks exhibit a lower BMD.

翻译：近期研究表明，神经网络的泛化误差存在双下降现象，即高度过参数化模型能够规避过拟合并取得优异测试性能，这与统计学习理论描述的标准偏差-方差权衡相悖。本研究探索了该现象与神经网络表征函数复杂度和灵敏度增加之间的关联。具体而言，我们研究了布尔函数分析领域提出的布尔平均维度（BMD）指标。聚焦于随机特征模型的简单师生学习框架，我们基于副本方法进行了理论分析，在数据点数量、特征维度和输入规模同步趋近无穷的高维极限下，推导出BMD的可解释表达式。研究发现，随着网络过参数化程度提升，BMD在内插阈值处出现显著峰值（与泛化误差峰值对应），而后缓慢趋近于低渐近值。这一现象在不同模型类别和训练设置的数值实验中得到验证。此外，实验发现对抗性初始化模型倾向于呈现更高BMD值，而对对抗攻击更具鲁棒性的模型则表现出更低BMD。