This technical report presents the application of a recurrent memory to extend the context length of BERT, one of the most effective Transformer-based models in natural language processing. By leveraging the Recurrent Memory Transformer architecture, we have successfully increased the model's effective context length to an unprecedented two million tokens, while maintaining high memory retrieval accuracy. Our method allows for the storage and processing of both local and global information and enables information flow between segments of the input sequence through the use of recurrence. Our experiments demonstrate the effectiveness of our approach, which holds significant potential to enhance long-term dependency handling in natural language understanding and generation tasks as well as enable large-scale context processing for memory-intensive applications.

翻译：本技术报告介绍了递归记忆在扩展BERT（自然语言处理中最有效的基于Transformer的模型之一）上下文长度中的应用。通过利用递归记忆Transformer架构，我们成功将模型的有效上下文长度扩展至前所未有的两百万个token，同时保持高记忆检索准确率。我们的方法能够存储和处理局部与全局信息，并通过循环机制实现输入序列各分段间的信息流通。实验证明，本方法的有效性为增强自然语言理解与生成任务中的长期依赖处理能力提供了重要潜力，同时能够支持对记忆密集型应用的大规模上下文处理。