Backpropagation (BP), the standard learning algorithm for artificial neural networks, is often considered biologically implausible. In contrast, the standard learning algorithm for predictive coding (PC) models in neuroscience, known as the inference learning algorithm (IL), is a promising, bio-plausible alternative. However, several challenges and questions hinder IL's application to real-world problems. For example, IL is computationally demanding, and without memory-intensive optimizers like Adam, IL may converge to poor local minima. Moreover, although IL can reduce loss more quickly than BP, the reasons for these speedups or their robustness remains unclear. In this paper, we tackle these challenges by 1) altering the standard implementation of PC circuits to substantially reduce computation, 2) developing a novel optimizer that improves the convergence of IL without increasing memory usage, and 3) establishing theoretical results that help elucidate the conditions under which IL is sensitive to second and higher-order information.

翻译：反向传播（BP）作为人工神经网络的标准学习算法，常被认为在生物学上不具合理性。相比之下，神经科学中预测编码（PC）模型的标准学习算法——即推理学习算法（IL）——是一种有前景的、生物合理的替代方案。然而，若干挑战和问题阻碍了IL在实际问题中的应用。例如，IL计算负荷大，且若缺乏像Adam这类高内存占用的优化器，IL可能收敛至较差的局部极小值。此外，尽管IL能比BP更快降低损失，但这种加速的原因及其鲁棒性仍不明确。本文通过以下方式应对这些挑战：1）修改PC电路的标准化实现以大幅减少计算量，2）开发一种在不增加内存使用的情况下改善IL收敛性的新型优化器，3）建立理论结果以阐明IL对二阶及更高阶信息敏感的条件。