Learning in neural networks is often framed as a problem in which targeted error signals are directly propagated to parameters and used to produce updates that induce more optimal network behaviour. Backpropagation of error (BP) is an example of such an approach and has proven to be a highly successful application of stochastic gradient descent to deep neural networks. We propose constrained parameter inference (COPI) as a new principle for learning. The COPI approach assumes that learning can be set up in a manner where parameters infer their own values based upon observations of their local neuron activities. We find that this estimation of network parameters is possible under the constraints of decorrelated neural inputs and top-down perturbations of neural states for credit assignment. We show that the decorrelation required for COPI allows learning at extremely high learning rates, competitive with that of adaptive optimizers, as used by BP. We further demonstrate that COPI affords a new approach to feature analysis and network compression. Finally, we argue that COPI may shed new light on learning in biological networks given the evidence for decorrelation in the brain.

翻译：神经网络中的学习常被表述为这样一种问题：目标误差信号直接传播到参数，并用于产生促使网络行为更优化的更新。误差反向传播（BP）便是此类方法的一个例子，它已被证明是随机梯度下降在深度神经网络中极为成功的应用。我们提出受约束参数推断（COPI）作为一种新的学习原则。COPI方法假设，学习可以这样设置：参数基于对其局部神经元活动的观察来推断自身的值。我们发现，在去相关神经输入和用于信用分配的自顶向下的神经状态扰动的约束下，这种网络参数的估计是可能的。我们表明，COPI所需的去相关允许以极高的学习率进行学习，其性能与BP所使用的自适应优化器相当。我们进一步证明，COPI为特征分析和网络压缩提供了一种新方法。最后，鉴于大脑中存在去相关的证据，我们认为COPI可能为生物网络中的学习提供新的见解。