The inductive bias of a neural network is largely determined by the architecture and the training algorithm. To achieve good generalization, how to effectively train a neural network is of great importance. We propose a novel orthogonal over-parameterized training (OPT) framework that can provably minimize the hyperspherical energy which characterizes the diversity of neurons on a hypersphere. By maintaining the minimum hyperspherical energy during training, OPT can greatly improve the network generalization. Specifically, OPT fixes the randomly initialized weights of the neurons and learns an orthogonal transformation that applies to these neurons. We propose multiple ways to learn such an orthogonal transformation, including unrolling orthogonalization algorithms, applying orthogonal parameterization, and designing orthogonality-preserving gradient descent. Interestingly, OPT reveals that learning a proper coordinate system for neurons is crucial to generalization and may be more important than learning specific relative positions among neurons. We provide some insights on why OPT yields better generalization. Extensive experiments validate the superiority of OPT.

翻译：神经网络的感知偏差在很大程度上由结构与培训算法决定。 为了实现良好的概括化, 如何有效地培训神经网络非常重要。 我们提出一个新的正统超分度培训框架, 它可以将超视球能量最小化, 即高视线神经多样性的特点。 通过在训练期间保持最低限度的超球能量, 巴勒斯坦被占领土可以大大改进网络的概括化。 具体地说, 巴勒斯坦被占领土可以修正神经元的随机初始重量, 并学习适用于这些神经元的正方形变异。 我们提出多种方法来学习这种正正方形变异, 包括不滚动或正方形变异算法, 应用正方形参数化法, 设计正方形梯度梯度下降。 有趣的是, 巴勒斯坦被占领土通过在训练中保持神经元的适当协调系统对于概括化至关重要, 并且可能比学习神经元之间特定相对位置更重要。 我们提供了一些关于为什么ALM产生更好的概括化的见解。