The performance of Large Language Models (LLMs) degrades from the temporal drift between data used for model training and newer text seen during inference. One understudied avenue of language change causing data drift is the emergence of neologisms -- new word forms -- over time. We create a diverse resource of recent English neologisms by using several popular collection methods. We analyze temporal drift using neologisms by comparing sentences containing new words with near-identical sentences that replace neologisms with existing substitute words. Model performance is nearly halved in machine translation when a single neologism is introduced in a sentence. Motivated by these results, we construct a benchmark to evaluate LLMs' ability to generalize to neologisms with various natural language understanding tasks and model perplexity. Models with later knowledge cutoff dates yield lower perplexities and perform better in downstream tasks. LLMs are also affected differently based on the linguistic origins of words, indicating that neologisms are complex for static LLMs to address. We will release our benchmark and code for reproducing our experiments.

翻译：大语言模型（LLM）的性能因模型训练所用数据与推理时所见新文本之间的时间漂移而下降。语言变化导致数据漂移的一个尚未充分研究的途径是新词（即新的单词形式）随时间涌现的过程。我们通过使用多种流行收集方法，创建了一个近期英语新词的多样化资源。我们利用新词分析时间漂移，方法是比较包含新词的句子与用现有替代词替换新词后近乎相同的句子。当句子中引入单个新词时，机器翻译中的模型性能几乎减半。受这些结果启发，我们构建了一个基准测试，通过多种自然语言理解任务和模型困惑度来评估LLM对新词的泛化能力。知识截止日期较晚的模型展现出更低的困惑度，并在下游任务中表现更优。此外，LLM受新词影响的程度因词语的语言来源不同而异，这表明对于静态LLM而言，新词问题具有复杂性。我们将发布我们的基准测试和用于复现实验的代码。