Language Models (LMs) become outdated as the world changes; they often fail to perform tasks requiring recent factual information which was absent or different during training, a phenomenon called temporal misalignment. This is especially a challenging problem because the research community still lacks a coherent dataset for assessing the adaptability of LMs to frequently-updated knowledge corpus such as Wikipedia. To this end, we introduce TemporalWiki, a lifelong benchmark for ever-evolving LMs that utilizes the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation, respectively. The benchmark hence allows researchers to periodically track an LM's ability to retain previous knowledge and acquire updated/new knowledge at each point in time. We also find that training an LM on the diff data through continual learning methods achieves similar or better perplexity than on the entire snapshot in our benchmark with 12 times less computational cost, which verifies that factual knowledge in LMs can be safely updated with minimal training data via continual learning. The dataset and the code are available at https://github.com/joeljang/temporalwiki.

翻译：语言模型（LM）会随世界变化而过时；它们往往无法完成需要近期事实信息的任务，这些信息在训练时缺失或不同，这一现象称为时间错位。这是一个特别棘手的问题，因为研究界仍缺乏一个连贯的数据集来评估LM对频繁更新的知识库（如维基百科）的适应能力。为此，我们提出了TemporalWiki，这是一个面向持续演化LM的终身基准，利用英文维基百科和英文维基数据连续快照之间的差异分别进行训练和评估。该基准因此允许研究人员定期追踪LM在任意时间点保留已有知识和获取更新/新知识的能力。我们还发现，通过持续学习方法在差异数据上训练LM，其困惑度与在基准中完整快照上训练相似或更优，而计算成本降低12倍，这证实了通过持续学习可用最小训练数据安全更新LM中的事实知识。数据集和代码可在 https://github.com/joeljang/temporalwiki 获取。