In the universe of Natural Language Processing, Transformer-based language models like BERT and (Chat)GPT have emerged as lexical superheroes with great power to solve open research problems. In this paper, we specifically focus on the temporal problem of semantic change, and evaluate their ability to solve two diachronic extensions of the Word-in-Context (WiC) task: TempoWiC and HistoWiC. In particular, we investigate the potential of a novel, off-the-shelf technology like ChatGPT (and GPT) 3.5 compared to BERT, which represents a family of models that currently stand as the state-of-the-art for modeling semantic change. Our experiments represent the first attempt to assess the use of (Chat)GPT for studying semantic change. Our results indicate that ChatGPT performs significantly worse than the foundational GPT version. Furthermore, our results demonstrate that (Chat)GPT achieves slightly lower performance than BERT in detecting long-term changes but performs significantly worse in detecting short-term changes.

翻译：在自然语言处理的宇宙中，基于Transformer的语言模型（如BERT和(Chat)GPT）如同拥有巨大力量的词汇超级英雄，能够解决开放性研究问题。本文聚焦语义变化这一时间性难题，评估它们在解决两种词汇上下文（WiC）任务历史扩展版——TempoWiC和HistoWiC——方面的能力。具体而言，我们探究了ChatGPT（及GPT 3.5）这类新型现成技术相比BERT的潜力——BERT代表当前语义变化建模领域最先进的模型家族。我们的实验首次尝试评估(Chat)GPT在语义变化研究中的应用。结果表明，ChatGPT的表现显著逊色于基础GPT版本。此外，实验结果证明，(Chat)GPT在检测长期语义变化时表现略低于BERT，但在检测短期变化时表现显著更差。