Large Language Models (LLMs) have become a mainstay for many everyday applications. However, as data evolve their knowledge quickly becomes outdated. Continual learning aims to update LLMs with new information without erasing previously acquired knowledge. Although methods such as full fine-tuning can incorporate new data, they are computationally expensive and prone to catastrophic forgetting, where prior knowledge is overwritten. Memory-augmented approaches address this by equipping LLMs with a memory bank, that is an external memory module which stores information for future use. However, these methods face a critical limitation, in particular, the memory bank constantly grows in the real-world scenario when large-scale data streams arrive. In this paper, we propose MBC, a model that compresses the memory bank through a codebook optimization strategy during online adaptation learning. To ensure stable learning, we also introduce an online resetting mechanism that prevents codebook collapse. In addition, we employ Key-Value Low-Rank Adaptation in the attention layers of the LLM, enabling efficient utilization of the compressed memory representations. Experiments with benchmark question-answering datasets demonstrate that MBC reduces the memory bank size to 0.3% when compared against the most competitive baseline, while maintaining high retention accuracy during online adaptation learning. Our code is publicly available at https://github.com/Thomkat/MBC.

翻译：大型语言模型已成为许多日常应用的主流技术。然而，随着数据演变，其知识会迅速过时。持续学习旨在用新信息更新大型语言模型，同时不抹除先前获得的知识。尽管全参数微调等方法能够整合新数据，但其计算成本高昂且容易发生灾难性遗忘，即先前知识被覆盖。基于记忆增强的方法通过为大型语言模型配备记忆库来解决这一问题，记忆库是一个存储信息供未来使用的外部记忆模块。然而，这些方法面临一个关键限制：在现实场景中，当大规模数据流持续到达时，记忆库会不断增长。本文提出MBC模型，该模型通过在线适应学习期间的码本优化策略压缩记忆库。为确保稳定学习，我们还引入了在线重置机制以防止码本坍缩。此外，我们在大型语言模型的注意力层采用键值低秩适应技术，从而实现对压缩记忆表征的高效利用。在基准问答数据集上的实验表明，与最具竞争力的基线相比，MBC将记忆库大小减少至0.3%，同时在线适应学习期间保持较高的记忆保留准确率。我们的代码已在https://github.com/Thomkat/MBC公开。