Predictive coding (PC) is a brain-inspired local learning algorithm that has recently been suggested to provide advantages over backpropagation (BP) in biologically relevant scenarios. While theoretical work has mainly focused on showing how PC can approximate BP in various limits, the putative benefits of "natural" PC are less understood. Here we develop a theory of PC as an adaptive trust-region (TR) algorithm that uses second-order information. We show that the learning dynamics of PC can be interpreted as interpolating between BP's loss gradient direction and a TR direction found by the PC inference dynamics. Our theory suggests that PC should escape saddle points faster than BP, a prediction which we prove in a shallow linear model and support with experiments on deeper networks. This work lays a foundation for understanding PC in deep and wide networks.

翻译：预测编码（PC）是一种受大脑启发的局部学习算法，近年来被认为在生物相关场景中比反向传播（BP）具有优势。虽然理论工作主要集中于展示PC如何在各种极限条件下近似BP，但“自然”PC的假定优势仍未被充分理解。本文发展了PC作为利用二阶信息的自适应信任域（TR）算法的理论。我们证明，PC的学习动态可被解释为在BP的损失梯度方向与PC推理动态所发现的信任域方向之间进行插值。我们的理论表明，PC应比BP更快逃离鞍点——我们在浅层线性模型中证明了这一预测，并通过更深层网络的实验予以支持。本工作为理解深层与宽层网络中的PC奠定了基础。