Good weight initialisation is an important step in successful training of Artificial Neural Networks. Over time a number of improvements have been proposed to this process. In this paper we introduce a novel weight initialisation technique called the Straddled Matrix Initialiser. This initialisation technique is motivated by our assumption that major, global-scale relationships in data are linear with only smaller effects requiring complex non-linearities. Combination of Straddled Matrix and ReLU activation function initialises a Neural Network as a de facto linear model, which we postulate should be a better starting point for optimisation given our assumptions. We test this by training autoencoders on three datasets using Straddled Matrix and seven other state-of-the-art weight initialisation techniques. In all our experiments the Straddeled Matrix Initialiser clearly outperforms all other methods.

翻译：良好的权重初始化是成功训练人工神经网络的关键步骤。随着时间的推移，该过程已涌现出多项改进方法。本文提出一种名为跨步矩阵初始化器的新型权重初始化技术。该初始化技术的动机源于我们的假设：数据中的主要全局尺度关系是线性的，仅需少量非线性效应处理局部细节。跨步矩阵与ReLU激活函数的组合将神经网络初始化为一个事实上的线性模型，我们推测基于该假设，此起点应更有利于优化过程。为验证这一假设，我们在三个数据集上使用跨步矩阵及其他七种前沿权重初始化技术训练自编码器。所有实验结果表明，跨步矩阵初始化器均显著优于其他方法。