Transformers are considered conceptually different compared to the previous generation of state-of-the-art NLP models - recurrent neural networks (RNNs). In this work, we demonstrate that decoder-only transformers can in fact be conceptualized as infinite multi-state RNNs - an RNN variant with unlimited hidden state size. We further show that pretrained transformers can be converted into $\textit{finite}$ multi-state RNNs by fixing the size of their hidden state. We observe that several existing transformers cache compression techniques can be framed as such conversion policies, and introduce a novel policy, TOVA, which is simpler compared to these policies. Our experiments with several long range tasks indicate that TOVA outperforms all other baseline policies, while being nearly on par with the full (infinite) model, and using in some cases only $\frac{1}{8}$ of the original cache size. Our results indicate that transformer decoder LLMs often behave in practice as RNNs. They also lay out the option of mitigating one of their most painful computational bottlenecks - the size of their cache memory. We publicly release our code at https://github.com/schwartz-lab-NLP/TOVA.

翻译：Transformer被视为与上一代最先进的自然语言处理模型——循环神经网络（RNN）——在概念上存在本质差异。本研究证明，仅解码器结构的Transformer实际上可被理解为无限多状态RNN——一种隐藏状态规模不受限制的RNN变体。我们进一步表明，通过固定隐藏状态尺寸，可将预训练Transformer转化为$\textit{有限}$多状态RNN。同时发现，现有若干Transformer缓存压缩技术可被归类为此类转化策略，并由此提出一种比这些策略更简洁的新型方法TOVA。我们在多个长程任务上的实验表明，TOVA在部分情况下仅使用原始缓存大小的$\frac{1}{8}$，不仅全面超越其他基线策略，且性能几乎与完整（无限）模型持平。实验结果揭示，Transformer解码器式大语言模型在实践中常展现出RNN特性，同时为缓解其最棘手的计算瓶颈之一——缓存内存规模——提供了可行性方案。我们已在https://github.com/schwartz-lab-NLP/TOVA 公开发布代码。