We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Transformer with sliding-window attention. However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights. In addition, we improve the model's initialization for learning at test time via meta-learning at training time. Overall, our method, a form of Test-Time Training (TTT), is End-to-End (E2E) both at test time (via next-token prediction) and training time (via meta-learning), in contrast to previous forms. We conduct extensive experiments with a focus on scaling properties. In particular, for 3B models trained with 164B tokens, our method (TTT-E2E) scales with context length in the same way as Transformer with full attention, while others, such as Mamba 2 and Gated DeltaNet, do not. However, similar to RNNs, TTT-E2E has constant inference latency regardless of context length, making it 2.7 times faster than full attention for 128K context. Our code is publicly available.

翻译：我们将长上下文语言建模重新表述为一个持续学习问题，而非架构设计问题。在此框架下，我们仅采用标准架构——即配备滑动窗口注意力的Transformer。然而，我们的模型在测试时通过基于给定上下文的下一个词元预测持续学习，将其读取的上下文信息压缩至权重中。此外，我们通过在训练阶段进行元学习，优化模型在测试时学习的初始化状态。总体而言，我们的方法作为一种测试时训练形式，在测试时（通过下一个词元预测）和训练时（通过元学习）均实现端到端处理，与此前的方法形成对比。我们进行了大量实验，重点关注缩放特性。具体而言，对于使用1640亿词元训练的30亿参数模型，我们的方法在上下文长度缩放方面表现出与完全注意力Transformer相同的特性，而其他方法（如Mamba 2和Gated DeltaNet）则未能实现。然而，与循环神经网络类似，我们的方法具有不随上下文长度变化的恒定推理延迟，在处理128K上下文时比完全注意力机制快2.7倍。相关代码已公开提供。