Large language models encode surprisingly broad knowledge about the world into their parameters. However, the knowledge in static language models can fall out of date, limiting the model's effective "shelf life." While online fine-tuning can reduce this degradation, we find that fine-tuning on a stream of documents using standard optimizers such as Adam leads to a disappointingly low level of information uptake. We hypothesize that online fine-tuning does not sufficiently 'attend' to important information. That is, the gradient signal from important tokens representing factual information is drowned out by the gradient from inherently noisy tokens, suggesting a dynamic, context-aware learning rate may be beneficial. To test this hypothesis, we meta-train a small, autoregressive model to reweight the language modeling loss for each token during online fine-tuning, with the objective of maximizing the out-of-date base language model's ability to answer questions about a document after a single weighted gradient step. We call this approach Context-aware Meta-learned Loss Scaling (CaMeLS). Across three different distributions of documents, our experiments find that fine-tuning on streams of thousands of documents with CaMeLS substantially improves knowledge retention compared to standard online fine-tuning. Finally, we find that the meta-learned weights are general, and that a single reweighting model can be used to enhance the online adaptation of many LMs.

翻译：摘要：大型语言模型在其参数中编码了关于世界的广泛知识。然而，静态语言模型中的知识可能过时，限制了模型的有效“保质期”。虽然在线微调可以缓解这一退化，但我们发现，使用Adam等标准优化器对文档流进行微调会导致信息吸收率低得令人失望。我们假设在线微调未能充分“关注”重要信息。也就是说，代表事实信息的重要标记的梯度信号被固有噪声标记的梯度淹没，这表明动态、上下文感知的学习率可能是有益的。为验证这一假设，我们元训练一个小型自回归模型，在在线微调期间对每个标记的语言建模损失进行重新加权，目标是在单次加权梯度更新后，最大化过时的基础语言模型回答文档中问题的能力。我们将这种方法称为上下文感知元学习损失缩放（CaMeLS）。在三种不同的文档分布中，我们的实验发现，与标准在线微调相比，使用CaMeLS在数千个文档流上进行微调显著提高了知识保留率。最后，我们发现元学习权重具有通用性，单个重新加权模型可用于增强多个语言模型的在线适应能力。