Amidst the sharp rise in the evaluation of large language models (LLMs) on various tasks, we find that semantic textual similarity (STS) has been under-explored. In this study, we show that STS can be cast as a text generation problem while maintaining strong performance on multiple STS benchmarks. Additionally, we show generative LLMs significantly outperform existing encoder-based STS models when characterizing the semantic similarity between two texts with complex semantic relationships dependent on world knowledge. We validate this claim by evaluating both generative LLMs and existing encoder-based STS models on three newly collected STS challenge sets which require world knowledge in the domains of Health, Politics, and Sports. All newly collected data is sourced from social media content posted after May 2023 to ensure the performance of closed-source models like ChatGPT cannot be credited to memorization. Our results show that, on average, generative LLMs outperform the best encoder-only baselines by an average of 22.3% on STS tasks requiring world knowledge. Our results suggest generative language models with STS-specific prompting strategies achieve state-of-the-art performance in complex, domain-specific STS tasks.

翻译：在大语言模型（LLMs）各类任务评估激增的背景下，我们发现语义文本相似度（STS）任务尚未得到充分探索。本研究证明，STS可转化为文本生成问题，并在多个STS基准测试中保持强劲性能。此外，我们表明，在刻画需要世界知识的复杂语义关系文本对之间的语义相似度时，生成式LLMs显著优于现有基于编码器的STS模型。通过在健康、政治和体育三个领域新构建的需世界知识的STS挑战集上对生成式LLMs及现有基于编码器的STS模型进行评估，我们验证了这一论断。所有新采集数据均源自2023年5月后发布的社交媒体内容，以确保ChatGPT等闭源模型的性能无法归因于记忆效应。结果表明，在需要世界知识的STS任务中，生成式LLMs平均性能比最优纯编码器基线提升22.3%。我们的结论表明，采用STS专用提示策略的生成式语言模型在复杂领域特定STS任务中达到了最先进水平。