The storage capacity of a binary classification model is the maximum number of random input-output pairs per parameter that the model can learn. It is one of the indicators of the expressive power of machine learning models and is important for comparing the performance of various models. In this study, we analyze the structure of the solution space and the storage capacity of fully connected two-layer neural networks with general activation functions using the replica method from statistical physics. Our results demonstrate that the storage capacity per parameter remains finite even with infinite width and that the weights of the network exhibit negative correlations, leading to a 'division of labor'. In addition, we find that increasing the dataset size triggers a phase transition at a certain transition point where the permutation symmetry of weights is broken, resulting in the solution space splitting into disjoint regions. We identify the dependence of this transition point and the storage capacity on the choice of activation function. These findings contribute to understanding the influence of activation functions and the number of parameters on the structure of the solution space, potentially offering insights for selecting appropriate architectures based on specific objectives.

翻译：二分类模型的存储容量是指模型每个参数所能学习的最大随机输入-输出对数量，它是衡量机器学习模型表达能力的重要指标之一，对比较不同模型的性能具有重要意义。本研究运用统计物理中的复制方法，分析了具有通用激活函数的全连接双层神经网络的解空间结构及存储容量。结果表明，即使在无限宽度下，每参数的存储容量仍保持有限，且网络权重呈现负相关，导致"劳动分工"现象。此外，我们发现数据集规模增大会在特定转变点触发相变，此时权重的排列对称性被打破，解空间分裂为互不相交的区域。我们确定了这一转变点及存储容量对激活函数选择的依赖性。这些发现有助于理解激活函数和参数数量对解空间结构的影响，或可为根据特定目标选择适当网络架构提供启示。