Language Models (LMs) become outdated as the world changes; they often fail to perform tasks requiring recent factual information which was absent or different during training, a phenomenon called temporal misalignment. This is especially a challenging problem because the research community still lacks a coherent dataset for assessing the adaptability of LMs to frequently-updated knowledge corpus such as Wikipedia. To this end, we introduce TemporalWiki, a lifelong benchmark for ever-evolving LMs that utilizes the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation, respectively. The benchmark hence allows researchers to periodically track an LM's ability to retain previous knowledge and acquire updated/new knowledge at each point in time. We also find that training an LM on the diff data through continual learning methods achieves similar or better perplexity than on the entire snapshot in our benchmark with 12 times less computational cost, which verifies that factual knowledge in LMs can be safely updated with minimal training data via continual learning. The dataset and the code are available at https://github.com/joeljang/temporalwiki.

翻译：语言模型（LMs）会随着世界变化而过时，难以执行需要近期事实信息的任务——这些信息在训练时可能缺失或存在差异，这一现象称为时间失配。该问题尤为棘手，因为研究界仍缺乏用于评估语言模型对频繁更新的知识语料库（如维基百科）适应能力的连贯数据集。为此，我们提出TemporalWiki——面向持续演化语言模型的终身基准，通过利用英语维基百科和维基数据连续快照之间的差异，分别进行训练与评估。该基准使研究者能够定期追踪语言模型在不同时间点保留旧知识、获取更新/新知识的能力。我们还发现，通过持续学习方法对差异数据进行训练，在基准测试中能以12倍的计算成本降低达到与全快照训练相当或更优的困惑度，这证实了通过持续学习用极少量训练数据即可安全更新语言模型中的事实知识。数据集与代码已开源至https://github.com/joeljang/temporalwiki。