Long-term memory is a cornerstone of human intelligence. Enabling AI to process lifetime-scale information remains a long-standing pursuit in the field. Due to the constraints of full-attention architectures, the effective context length of large language models (LLMs) is typically limited to 1M tokens. Existing approaches, such as hybrid linear attention, fixed-size memory states (e.g., RNNs), and external storage methods like RAG or agent systems, attempt to extend this limit. However, they often suffer from severe precision degradation and rapidly increasing latency as context length grows, an inability to dynamically modify memory content, or a lack of end-to-end optimization. These bottlenecks impede complex scenarios like large-corpus summarization, Digital Twins, and long-history agent reasoning, while limiting memory capacity and slowing inference. We present Memory Sparse Attention (MSA), an end-to-end trainable, efficient, and massively scalable memory model framework. Through core innovations including scalable sparse attention and document-wise RoPE, MSA achieves linear complexity in both training and inference while maintaining exceptional stability, exhibiting less than 9% degradation when scaling from 16K to 100M tokens. Furthermore, KV cache compression, combined with Memory Parallel, enables 100M-token inference on 2xA800 GPUs. We also propose Memory Interleaving to facilitate complex multi-hop reasoning across scattered memory segments. MSA significantly surpasses frontier LLMs, state-of-the-art RAG systems, and leading memory agents in long-context benchmarks. These results demonstrate that by decoupling memory capacity from reasoning, MSA provides a scalable foundation to endow general-purpose models with intrinsic, lifetime-scale memory.

翻译：长期记忆是人类智能的基石。使人工智能能够处理生命周期规模的信息一直是该领域的长期追求。由于全注意力架构的限制，大语言模型的有效上下文长度通常限于100万令牌。现有方法，如混合线性注意力、固定大小记忆状态（如循环神经网络）和外部存储方法（如RAG或智能体系统），试图突破这一限制。然而，这些方法通常面临随着上下文长度增长而出现的严重精度下降和快速增加的延迟、无法动态修改记忆内容，或缺乏端到端优化等问题。这些瓶颈阻碍了大规模语料摘要、数字孪生和长历史智能体推理等复杂场景的实现，同时限制了记忆容量并降低了推理速度。我们提出了记忆稀疏注意力（MSA），这是一种端到端可训练、高效且可大规模扩展的记忆模型框架。通过可扩展稀疏注意力和文档级旋转位置编码等核心创新，MSA在训练和推理中实现了线性复杂度，同时保持卓越的稳定性，当从16K令牌扩展到1亿令牌时性能下降不到9%。此外，结合KV缓存压缩与记忆并行技术，可在2块A800 GPU上实现1亿令牌推理。我们还提出了记忆交织技术以促进跨分散记忆片段的复杂多跳推理。MSA在长上下文基准测试中显著超越了前沿大语言模型、最先进的RAG系统和领先的记忆智能体。这些结果表明，通过将记忆容量与推理解耦，MSA为赋予通用模型固有的大规模长期记忆提供了可扩展的基础。