Large language models (LLMs) are increasingly capable of completing knowledge intensive tasks by recalling information from a static pretraining corpus. Here we are concerned with LLMs in the context of evolving data requirements. For instance: batches of new data that are introduced periodically; subsets of data with user-based access controls; or requirements on dynamic removal of documents with guarantees that associated knowledge cannot be recalled. We wish to satisfy these requirements while at the same time ensuring a model does not forget old information when new data becomes available. To address these issues, we introduce AdapterSwap, a training and inference scheme that organizes knowledge from a data collection into a set of low-rank adapters, which are dynamically composed during inference. Our experiments demonstrate AdapterSwap's ability to support efficient continual learning, while also enabling organizations to have fine-grained control over data access and deletion.

翻译：大型语言模型（LLM）日益能够通过从静态预训练语料库中回忆信息来完成知识密集型任务。本文关注于LLM在数据需求不断变化场景下的应用，例如：定期引入的新数据批次、基于用户访问控制的数据子集、或对文档动态删除且保证无法回忆相关知识的要求。我们希望在满足这些需求的同时，确保模型在获取新数据时不遗忘已有信息。为此，我们提出AdapterSwap——一种训练与推理方案，它将数据集合中的知识组织为一组低秩适配器，并在推理过程中动态组合。实验表明，AdapterSwap能够支持高效的持续学习，同时使组织对数据访问与删除实现细粒度控制。