Transformers have been established as the de-facto backbones for most recent advances in sequence modeling, mainly due to their growing memory capacity that scales with the context length. While plausible for retrieval tasks, it causes quadratic complexity and so has motivated recent studies to explore viable subquadratic recurrent alternatives. Despite showing promising preliminary results in diverse domains, such recurrent architectures underperform Transformers in recall-intensive tasks, often attributed to their fixed-size memory. In this paper, we introduce Memory Caching (MC), a simple yet effective technique that enhances recurrent models by caching checkpoints of their memory states (a.k.a. hidden states). Memory Caching allows the effective memory capacity of RNNs to grow with sequence length, offering a flexible trade-off that interpolates between the fixed memory (i.e., $O(L)$ complexity) of RNNs and the growing memory (i.e., $O(L^2)$ complexity) of Transformers. We propose four variants of MC, including gated aggregation and sparse selective mechanisms, and discuss their implications on both linear and deep memory modules. Our experimental results on language modeling, and long-context understanding tasks show that MC enhances the performance of recurrent models, supporting its effectiveness. The results of in-context recall tasks indicate that while Transformers achieve the best accuracy, our MC variants show competitive performance, close the gap with Transformers, and performs better than state-of-the-art recurrent models.

翻译：Transformer凭借其随上下文长度扩展而增长的内存容量，已成为当前序列建模领域绝大多数进展的事实基础架构。尽管这对于检索任务而言是合理的，但它带来了二次复杂度，从而促使近期研究探索可行的次二次循环替代方案。尽管此类循环架构在多个领域展现出有前景的初步结果，但在召回密集型任务中表现仍逊于Transformer，这通常归因于其固定大小的记忆容量。本文提出记忆缓存（Memory Caching，MC），这是一种简单而有效的技术，通过缓存循环模型记忆状态（即隐藏状态）的检查点来增强其性能。记忆缓存使循环神经网络的有效记忆容量能够随序列长度增长，提供了一种灵活的权衡方案，在循环神经网络的固定记忆（即$O(L)$复杂度）与Transformer的增长记忆（即$O(L^2)$复杂度）之间实现插值。我们提出了MC的四种变体，包括门控聚合和稀疏选择机制，并讨论了它们对线性和深度记忆模块的影响。我们在语言建模和长上下文理解任务上的实验结果表明，MC能有效提升循环模型的性能。在上下文召回任务中，虽然Transformer取得了最佳准确率，但我们的MC变体展现出具有竞争力的性能，缩小了与Transformer的差距，并优于当前最先进的循环模型。