Heuristic tools from statistical physics have been used in the past to locate the phase transitions and compute the optimal learning and generalization errors in the teacher-student scenario in multi-layer neural networks. In this contribution, we provide a rigorous justification of these approaches for a two-layers neural network model called the committee machine. We also introduce a version of the approximate message passing (AMP) algorithm for the committee machine that allows to perform optimal learning in polynomial time for a large set of parameters. We find that there are regimes in which a low generalization error is information-theoretically achievable while the AMP algorithm fails to deliver it, strongly suggesting that no efficient algorithm exists for those cases, and unveiling a large computational gap.

翻译：过去，统计物理中的启发式工具已被用于定位多层神经网络中教师-学生场景的相变，并计算最优学习与泛化误差。在本研究中，我们为一种名为委员会机器的双层神经网络模型提供了这些方法的严格论证。我们还针对委员会机器引入了一种近似消息传递（AMP）算法版本，该算法能够在大量参数设置下以多项式时间实现最优学习。我们发现存在某些机制，其中低泛化误差在信息论上是可达的，但AMP算法却无法实现，这强烈表明这些情况不存在高效算法，并揭示了一个巨大的计算差距。