In the universe of Natural Language Processing, Transformer-based language models like BERT and (Chat)GPT have emerged as lexical superheroes with great power to solve open research problems. In this paper, we specifically focus on the temporal problem of semantic change, and evaluate their ability to solve two diachronic extensions of the Word-in-Context (WiC) task: TempoWiC and HistoWiC. In particular, we investigate the potential of a novel, off-the-shelf technology like ChatGPT (and GPT) 3.5 compared to BERT, which represents a family of models that currently stand as the state-of-the-art for modeling semantic change. Our experiments represent the first attempt to assess the use of (Chat)GPT for studying semantic change. Our results indicate that ChatGPT performs significantly worse than the foundational GPT version. Furthermore, our results demonstrate that (Chat)GPT achieves slightly lower performance than BERT in detecting long-term changes but performs significantly worse in detecting short-term changes.

翻译：在自然语言处理领域中，基于Transformer的语言模型（如BERT和(Chat)GPT）如同词汇领域的超级英雄，凭借强大能力解决开放式研究问题。本文聚焦语义变化的时间维度问题，评估它们解决两种历时扩展的词语上下文（WiC）任务——TempoWiC和HistoWiC的能力。具体而言，我们探究了ChatGPT（及GPT）3.5这类新型现成技术与当前语义变化建模领域最先进模型家族BERT的对比潜力。我们的实验首次尝试评估(Chat)GPT在语义变化研究中的应用。结果表明，ChatGPT的表现显著弱于基础GPT版本。此外，实验证明(Chat)GPT在检测长期变化时性能略低于BERT，但在检测短期变化时表现显著更差。