Semantic textual similarity (STS), a cornerstone task in NLP, measures the degree of similarity between a pair of sentences, and has broad application in fields such as information retrieval and natural language understanding. However, sentence similarity can be inherently ambiguous, depending on the specific aspect of interest. We resolve this ambiguity by proposing a novel task called Conditional STS (C-STS) which measures sentences' similarity conditioned on an feature described in natural language (hereon, condition). As an example, the similarity between the sentences "The NBA player shoots a three-pointer." and "A man throws a tennis ball into the air to serve." is higher for the condition "The motion of the ball" (both upward) and lower for "The size of the ball" (one large and one small). C-STS's advantages are two-fold: (1) it reduces the subjectivity and ambiguity of STS and (2) enables fine-grained language model evaluation through diverse natural language conditions. We put several state-of-the-art models to the test, and even those performing well on STS (e.g. SimCSE, Flan-T5, and GPT-4) find C-STS challenging; all with Spearman correlation scores below 50. To encourage a more comprehensive evaluation of semantic similarity and natural language understanding, we make nearly 19K C-STS examples and code available for others to train and test their models.

翻译：摘要：语义文本相似度（STS）作为自然语言处理中的基础任务，用于衡量句子对之间的相似程度，并在信息检索和自然语言理解等领域具有广泛应用。然而，句子相似度存在固有歧义，其判定往往取决于所关注的特定层面。为解决这一歧义，我们提出了一项名为条件性语义文本相似度（C-STS）的新任务，该任务基于自然语言描述的特定特征（以下简称"条件"）来度量句子间的相似性。例如，对于句子"The NBA player shoots a three-pointer."和"A man throws a tennis ball into the air to serve."，在条件"球的运动轨迹"下（两者均为向上运动）相似度较高，而在条件"球的尺寸"下（一大一小）相似度较低。C-STS具有双重优势：（1）降低了STS的主观性与歧义性；（2）通过多样化的自然语言条件实现细粒度语言模型评估。我们对多个最先进的模型进行了测试，即便在STS任务上表现优异的模型（如SimCSE、Flan-T5和GPT-4）在C-STS中也面临挑战——所有模型的斯皮尔曼相关系数均低于50。为促进对语义相似度与自然语言理解更全面的评估，我们公开了约1.9万个C-STS样本及相应代码，供研究者用于模型训练与测试。