Due to the rapid generation and dissemination of information, large language models (LLMs) quickly run out of date despite enormous development costs. Due to this crucial need to keep models updated, online learning has emerged as a critical necessity when utilizing LLMs for real-world applications. However, given the ever-expanding corpus of unseen documents and the large parameter space of modern LLMs, efficient adaptation is essential. To address these challenges, we propose Memory of Amortized Contexts (MAC), an efficient and effective online adaptation framework for LLMs with strong knowledge retention. We propose an amortized feature extraction and memory-augmentation approach to compress and extract information from new documents into compact modulations stored in a memory bank. When answering questions, our model attends to and extracts relevant knowledge from this memory bank. To learn informative modulations in an efficient manner, we utilize amortization-based meta-learning, which substitutes the optimization process with a single forward pass of the encoder. Subsequently, we learn to choose from and aggregate selected documents into a single modulation by conditioning on the question, allowing us to adapt a frozen language model during test time without requiring further gradient updates. Our experiment demonstrates the superiority of MAC in multiple aspects, including online adaptation performance, time, and memory efficiency. Code is available at: https://github.com/jihoontack/MAC.

翻译：由于信息的快速生成与传播，大型语言模型（LLM）尽管开发成本巨大，却很快会过时。针对这一关键需求，在线学习已成为将LLM应用于实际场景时的必要手段。然而，面对不断扩展的未见文档语料库和现代LLM庞大的参数空间，高效适应显得至关重要。为解决这些挑战，我们提出了"摊销上下文记忆"（MAC）——一种兼具强知识保留能力的高效在线适应框架。我们设计了摊销特征提取与记忆增强方法，通过压缩新文档信息并将其转化为紧凑调制信号存入记忆库。在回答问题阶段，模型会从该记忆库中检索并提取相关知识。为实现高效学习信息调制信号，我们采用基于摊销的元学习技术，仅需编码器单次前向传播即可替代传统优化过程。随后，模型通过问题条件约束，学习从候选文档中选择并聚合生成单一调制信号，从而在测试阶段无需梯度更新即可适应冻结的语言模型。实验表明，MAC在在线适应性能、时间效率与内存效率等多维度均展现出优越性。代码开源地址：https://github.com/jihoontack/MAC。