We present a simple linear regression based approach for learning the weights and biases of a neural network, as an alternative to standard gradient based backpropagation. The present work is exploratory in nature, and we restrict the description and experiments to (i) simple feedforward neural networks, (ii) scalar (single output) regression problems, and (iii) invertible activation functions. However, the approach is intended to be extensible to larger, more complex architectures. The key idea is the observation that the input to every neuron in a neural network is a linear combination of the activations of neurons in the previous layer, as well as the parameters (weights and biases) of the layer. If we are able to compute the ideal total input values to every neuron by working backwards from the output, we can formulate the learning problem as a linear least squares problem which iterates between updating the parameters and the activation values. We present an explicit algorithm that implements this idea, and we show that (at least for simple problems) the approach is more stable and faster than gradient-based backpropagation.

翻译：我们提出了一种基于简单线性回归的方法来学习神经网络的权重和偏置，作为标准梯度反向传播的替代方案。本研究本质上属于探索性质，因此将描述和实验限制在：（i）简单的前馈神经网络；（ii）标量（单输出）回归问题；以及（iii）可逆激活函数。然而，该方法旨在可扩展到更大、更复杂的架构。关键思想在于观察到：神经网络中每个神经元的输入是前一层神经元激活值以及该层参数（权重和偏置）的线性组合。如果我们能够通过从输出反向推导来计算每个神经元的理想总输入值，就可以将学习问题表述为一个线性最小二乘问题，该问题在参数更新与激活值更新之间迭代进行。我们提出了一种实现该思想的显式算法，并证明（至少在简单问题上）该方法比基于梯度的反向传播更稳定且速度更快。