Equilibrium Propagation (EP) is a powerful and more bio-plausible alternative to conventional learning frameworks such as backpropagation. The effectiveness of EP stems from the fact that it relies only on local computations and requires solely one kind of computational unit during both of its training phases, thereby enabling greater applicability in domains such as bio-inspired neuromorphic computing. The dynamics of the model in EP is governed by an energy function and the internal states of the model consequently converge to a steady state following the state transition rules defined by the same. However, by definition, EP requires the input to the model (a convergent RNN) to be static in both the phases of training. Thus it is not possible to design a model for sequence classification using EP with an LSTM or GRU like architecture. In this paper, we leverage recent developments in modern hopfield networks to further understand energy based models and develop solutions for complex sequence classification tasks using EP while satisfying its convergence criteria and maintaining its theoretical similarities with recurrent backpropagation. We explore the possibility of integrating modern hopfield networks as an attention mechanism with convergent RNN models used in EP, thereby extending its applicability for the first time on two different sequence classification tasks in natural language processing viz. sentiment analysis (IMDB dataset) and natural language inference (SNLI dataset).

翻译：平衡传播（EP）是传统学习框架（如反向传播）的一种更强大且更具生物合理性的替代方案。EP的有效性源于其仅依赖局部计算，并且在两个训练阶段中只需使用同一种计算单元，从而在生物启发式神经形态计算等领域具有更强的适用性。EP中模型的动态由能量函数控制，模型内部状态根据该能量函数定义的状态转移规则收敛至稳态。然而，依据定义，EP要求模型（一个收敛的RNN）的输入在两个训练阶段中均保持静态。因此，无法使用EP设计基于LSTM或GRU架构的序列分类模型。本文利用现代Hopfield网络的最新进展，进一步理解基于能量的模型，并为使用EP解决复杂序列分类任务开发解决方案，同时满足其收敛条件并保持与循环反向传播的理论相似性。我们探索将现代Hopfield网络作为注意力机制与EP中使用的收敛RNN模型相结合的可能性，从而首次将其应用扩展到自然语言处理中的两个不同序列分类任务：情感分析（IMDB数据集）和自然语言推理（SNLI数据集）。